[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:11 dist-test-slave-cvwc krb5kdc[3063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:11 dist-test-slave-cvwc krb5kdc[3063](info): set up 2 sockets
Aug 11 02:37:11 dist-test-slave-cvwc krb5kdc[3063](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 02:37:12.573258 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:12 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:12.709403 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34307
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:12.803056 3083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:12.803315 3083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:12.803350 3083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:12.804684 3083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:12.804744 3083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:12.804762 3083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:12.804778 3083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:12.804793 3083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:12.806772 3083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:12.807240 3083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:12.807547 3083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:12.814900 3093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:12.814900 3090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:12.815492 3091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:12.816303 3083 server_base.cc:1034] running on GCE node
I20260811 02:37:12.816697 3083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:12.817123 3083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:12.818312 3083 hybrid_clock.cc:648] HybridClock initialized: now 1786415832818272 us; error 87 us; skew 500 ppm
Aug 11 02:37:12 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:12.821601 3083 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:12.822449 3083 webserver.cc:460] Webserver started at http://127.2.251.62:33535/ using document root <none> and password file <none>
I20260811 02:37:12.822855 3083 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:12.822940 3083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:12.823182 3083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:12.824409 3083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "7b189329ceab451ba5a3acd04f329843"
format_stamp: "Formatted at 2026-08-11 02:37:12 on dist-test-slave-cvwc"
server_key: "ae89d9b892700d7e852b00f23f62759f"
server_key_iv: "379cd6483a3add5923fbcd257a8665d6"
server_key_version: "encryptionkey@0"
I20260811 02:37:12.824800 3083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "7b189329ceab451ba5a3acd04f329843"
format_stamp: "Formatted at 2026-08-11 02:37:12 on dist-test-slave-cvwc"
server_key: "ae89d9b892700d7e852b00f23f62759f"
server_key_iv: "379cd6483a3add5923fbcd257a8665d6"
server_key_version: "encryptionkey@0"
I20260811 02:37:12.826748 3083 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 02:37:12.827826 3103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:12.828330 3083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:12.828469 3083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "7b189329ceab451ba5a3acd04f329843"
format_stamp: "Formatted at 2026-08-11 02:37:12 on dist-test-slave-cvwc"
server_key: "ae89d9b892700d7e852b00f23f62759f"
server_key_iv: "379cd6483a3add5923fbcd257a8665d6"
server_key_version: "encryptionkey@0"
I20260811 02:37:12.828563 3083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:12.846168 3083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:12.850457 3083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:12.850723 3083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:12.856637 3083 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:34307
I20260811 02:37:12.856741 3188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:34307 every 8 connection(s)
I20260811 02:37:12.857762 3083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:12.859475 3192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:12.862664 3192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843: Bootstrap starting.
I20260811 02:37:12.863449 3192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:12.863826 3192 log.cc:824] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:12.864888 3192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843: No bootstrap required, opened a new log
I20260811 02:37:12.866286 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3083
I20260811 02:37:12.866442 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:12.867074 3192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b189329ceab451ba5a3acd04f329843" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34307 } }
I20260811 02:37:12.867244 3192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:12.867338 3192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b189329ceab451ba5a3acd04f329843, State: Initialized, Role: FOLLOWER
I20260811 02:37:12.867427 3052 external_mini_cluster.cc:1428] Setting key 84a3f392b85a2754af012ad815485fb5
I20260811 02:37:12.867581 3192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b189329ceab451ba5a3acd04f329843" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34307 } }
I20260811 02:37:12.867705 3192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:12.867735 3192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:12.867787 3192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:12.868415 3192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b189329ceab451ba5a3acd04f329843" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34307 } }
I20260811 02:37:12.868635 3192 leader_election.cc:302] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b189329ceab451ba5a3acd04f329843; no voters:
I20260811 02:37:12.868880 3192 leader_election.cc:288] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:12.868907 3197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:12.869150 3197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [term 1 LEADER]: Becoming Leader. State: Replica: 7b189329ceab451ba5a3acd04f329843, State: Running, Role: LEADER
I20260811 02:37:12.869429 3197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b189329ceab451ba5a3acd04f329843" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34307 } }
I20260811 02:37:12.869434 3192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:12.870541 3203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b189329ceab451ba5a3acd04f329843. Latest consensus state: current_term: 1 leader_uuid: "7b189329ceab451ba5a3acd04f329843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b189329ceab451ba5a3acd04f329843" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34307 } } }
I20260811 02:37:12.870635 3203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:12.870961 3212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:12.871673 3204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b189329ceab451ba5a3acd04f329843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b189329ceab451ba5a3acd04f329843" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34307 } } }
I20260811 02:37:12.871835 3212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:12.871845 3204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:12.873971 3212 catalog_manager.cc:1347] Generated new cluster ID: 02ed68aee5164ea595c5f1d6d2529f88
I20260811 02:37:12.874027 3212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:12 dist-test-slave-cvwc krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:12.885408 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:12.868595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.2.251.62:34307)
0811 02:37:12.868907 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:12.868918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:12.869280 (+ 362us) server_negotiation.cc:407] Connection header received
0811 02:37:12.872128 (+ 2848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:12.872135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:12.872184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:12.872276 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:12.874609 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:12.876005 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:12.877300 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:12.877857 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:12.878010 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:12.881618 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:12.881623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:12.881623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:12.881634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:12.883401 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:12.883823 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:12.883825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:12.883826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:12.883910 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:12.884139 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:12.884140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:12.884140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:12.884399 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:12.884549 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:12.884897 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 02:37:12.885098 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260811 02:37:12.887324 3212 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:12.887910 3212 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:12.897336 3212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b189329ceab451ba5a3acd04f329843: Generated new TSK 0
I20260811 02:37:12.897691 3212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:12.942546 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34307
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:13.052865 3230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:13.053149 3230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:13.053180 3230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:13.054607 3230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:13.054673 3230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:13.054708 3230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:13.056828 3230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:13.057286 3230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:13.057554 3230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:13.063356 3236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:13.063481 3230 server_base.cc:1034] running on GCE node
W20260811 02:37:13.063351 3238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:13.063351 3240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:13.063798 3230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:13.064103 3230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:13.065356 3230 hybrid_clock.cc:648] HybridClock initialized: now 1786415833065307 us; error 48 us; skew 500 ppm
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:13.068872 3230 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:13.069363 3230 webserver.cc:460] Webserver started at http://127.2.251.1:45793/ using document root <none> and password file <none>
I20260811 02:37:13.069667 3230 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:13.069798 3230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:13.069986 3230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:13.071305 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "588b3c2e489343cc863851affb1d7954"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "89adf0f661d88428d85d11baec68612a"
server_key_iv: "bcec65dbf2be8c6aa8d100217c202cc5"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.071717 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "588b3c2e489343cc863851affb1d7954"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "89adf0f661d88428d85d11baec68612a"
server_key_iv: "bcec65dbf2be8c6aa8d100217c202cc5"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.073200 3230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:37:13.074321 3249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.074569 3230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:13.074673 3230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "588b3c2e489343cc863851affb1d7954"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "89adf0f661d88428d85d11baec68612a"
server_key_iv: "bcec65dbf2be8c6aa8d100217c202cc5"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.074744 3230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:13.083600 3230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:13.085942 3230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:13.086091 3230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:13.086400 3230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:13.086814 3230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:13.086905 3230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.086941 3230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:13.086966 3230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.093566 3230 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:35253
I20260811 02:37:13.093680 3414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:35253 every 8 connection(s)
I20260811 02:37:13.094079 3230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:13.100317 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3230
I20260811 02:37:13.100425 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:13.100595 3052 external_mini_cluster.cc:1428] Setting key a387dadc4bf2ae02f2773b90c6424b00
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:13.105842 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.095219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:36661 (local address 127.2.251.62:34307)
0811 02:37:13.095289 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:13.095291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:13.095547 (+ 256us) server_negotiation.cc:407] Connection header received
0811 02:37:13.096042 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:13.096043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:13.096056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:13.096119 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:13.097277 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.097848 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.098948 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.099175 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.099242 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.102060 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:13.102061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:13.102062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:13.102070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:13.103818 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.104388 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.104390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.104390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.104473 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.104820 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.104821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.104821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.104965 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:13.105032 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:13.105515 (+ 483us) server_negotiation.cc:299] Negotiation successful
0811 02:37:13.105667 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:37:13.105935 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.095117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34307 (local address 127.2.251.1:36661)
0811 02:37:13.095362 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:13.095426 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:13.095941 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:13.096179 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:13.096182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:13.096926 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:13.097150 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:13.097157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.097968 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.097970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:13.098830 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:13.098834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.099305 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.099307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:13.099449 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.100020 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:13.100030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:13.101944 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:13.103939 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:13.103953 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:13.103956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:13.104238 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:13.104552 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:13.104553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:13.104553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:13.104715 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:13.105036 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:13.105044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:13.105241 (+ 197us) client_negotiation.cc:769] Sending connection context
0811 02:37:13.105367 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 02:37:13.105582 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260811 02:37:13.106859 3415 heartbeater.cc:347] Connected to a master server at 127.2.251.62:34307
I20260811 02:37:13.106994 3415 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:13.107232 3415 heartbeater.cc:511] Master 127.2.251.62:34307 requested a full tablet report, sending...
I20260811 02:37:13.108394 3132 ts_manager.cc:194] Registered new tserver with Master: 588b3c2e489343cc863851affb1d7954 (127.2.251.1:35253)
I20260811 02:37:13.109407 3132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:36661
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:13.133224 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34307
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:13.223374 3425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:13.223613 3425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:13.223654 3425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:13.224958 3425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:13.225013 3425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:13.225054 3425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:13.226809 3425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:13.227106 3425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:13.227368 3425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:37:13.233361 3425 server_base.cc:1034] running on GCE node
W20260811 02:37:13.233374 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:13.233374 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:13.233510 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:13.233803 3425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:13.234081 3425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:13.235318 3425 hybrid_clock.cc:648] HybridClock initialized: now 1786415833235293 us; error 39 us; skew 500 ppm
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:13.238474 3425 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:13.238950 3425 webserver.cc:460] Webserver started at http://127.2.251.2:42995/ using document root <none> and password file <none>
I20260811 02:37:13.239207 3425 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:13.239254 3425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:13.239389 3425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:13.240558 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "424deb026b0d4657a117eb674a0961ec"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "2946f0a1940d695b41c56edf4f2edcd4"
server_key_iv: "aa6cc06beaa0be24d7e500ec72c34467"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.240908 3425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "424deb026b0d4657a117eb674a0961ec"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "2946f0a1940d695b41c56edf4f2edcd4"
server_key_iv: "aa6cc06beaa0be24d7e500ec72c34467"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.242465 3425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:37:13.243315 3451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.243526 3425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:13.243631 3425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "424deb026b0d4657a117eb674a0961ec"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "2946f0a1940d695b41c56edf4f2edcd4"
server_key_iv: "aa6cc06beaa0be24d7e500ec72c34467"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.243703 3425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:13.253543 3425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:13.256757 3425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:13.256937 3425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:13.257193 3425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:13.257531 3425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:13.257567 3425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.257597 3425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:13.257622 3425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.264492 3425 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:42601
I20260811 02:37:13.264570 3624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:42601 every 8 connection(s)
I20260811 02:37:13.264915 3425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:13.269791 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3425
I20260811 02:37:13.269896 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:13.270058 3052 external_mini_cluster.cc:1428] Setting key 036cda8bbe2743716bef44f56504f6fe
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:13.276345 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.265900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:57741 (local address 127.2.251.62:34307)
0811 02:37:13.265955 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:13.265957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:13.266200 (+ 243us) server_negotiation.cc:407] Connection header received
0811 02:37:13.266701 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:13.266703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:13.266715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:13.266797 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:13.267990 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.268506 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.269981 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.270331 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.270401 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.273159 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:13.273160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:13.273161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:13.273170 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:13.274787 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.275219 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.275221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.275221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.275338 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.275640 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.275641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.275641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.275779 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:13.275839 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:13.276133 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 02:37:13.276247 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:37:13.276525 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.265868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34307 (local address 127.2.251.2:57741)
0811 02:37:13.266043 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:13.266087 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:13.266596 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:13.266907 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:13.266912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:13.267607 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:13.267835 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:13.267840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.268616 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.268617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:13.269868 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:13.269872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.270416 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.270418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:13.270591 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.271122 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:13.271133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:13.273057 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:13.274884 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:13.274893 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:13.274896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:13.275145 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:13.275441 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:13.275442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:13.275442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:13.275575 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:13.275843 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:13.275851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:13.276005 (+ 154us) client_negotiation.cc:769] Sending connection context
0811 02:37:13.276102 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 02:37:13.276289 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260811 02:37:13.277063 3626 heartbeater.cc:347] Connected to a master server at 127.2.251.62:34307
I20260811 02:37:13.277164 3626 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:13.277372 3626 heartbeater.cc:511] Master 127.2.251.62:34307 requested a full tablet report, sending...
I20260811 02:37:13.277917 3132 ts_manager.cc:194] Registered new tserver with Master: 424deb026b0d4657a117eb674a0961ec (127.2.251.2:42601)
I20260811 02:37:13.278316 3132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:57741
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:13.301801 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34307
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:13.395339 3633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:13.395608 3633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:13.395637 3633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:13.396895 3633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:13.396950 3633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:13.396991 3633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:13.398864 3633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:13.399121 3633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:13.399331 3633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:13.405627 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:13.405690 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:13.405630 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:13.405659 3633 server_base.cc:1034] running on GCE node
I20260811 02:37:13.406467 3633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:13.406737 3633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:13.407930 3633 hybrid_clock.cc:648] HybridClock initialized: now 1786415833407917 us; error 33 us; skew 500 ppm
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:13.411293 3633 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:13.412000 3633 webserver.cc:460] Webserver started at http://127.2.251.3:46181/ using document root <none> and password file <none>
I20260811 02:37:13.412310 3633 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:13.412374 3633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:13.412683 3633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:13.414057 3633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "9fe395441d7a4e619e7bed21aef20d7f"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "32b9fe60dd3c24e49d1ce739367725d5"
server_key_iv: "f302ca173cbaa442b7b32f1e15d3feb2"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.414446 3633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "9fe395441d7a4e619e7bed21aef20d7f"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "32b9fe60dd3c24e49d1ce739367725d5"
server_key_iv: "f302ca173cbaa442b7b32f1e15d3feb2"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.416100 3633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:13.417346 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.417624 3633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:13.417778 3633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "9fe395441d7a4e619e7bed21aef20d7f"
format_stamp: "Formatted at 2026-08-11 02:37:13 on dist-test-slave-cvwc"
server_key: "32b9fe60dd3c24e49d1ce739367725d5"
server_key_iv: "f302ca173cbaa442b7b32f1e15d3feb2"
server_key_version: "encryptionkey@0"
I20260811 02:37:13.417888 3633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:13.434791 3633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:13.437508 3633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:13.437685 3633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:13.438122 3633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:13.438514 3633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:13.438555 3633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.438589 3633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:13.438606 3633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:13.445744 3633 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:40333
I20260811 02:37:13.445869 3828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:40333 every 8 connection(s)
I20260811 02:37:13.446379 3633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:13.449525 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3633
I20260811 02:37:13.449633 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:13.449841 3052 external_mini_cluster.cc:1428] Setting key 1893d44af7160eceb736cd131c5d0fff
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:13.458475 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.447736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:40935 (local address 127.2.251.62:34307)
0811 02:37:13.447799 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:13.447802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:13.448059 (+ 257us) server_negotiation.cc:407] Connection header received
0811 02:37:13.448695 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:13.448697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:13.448714 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:13.448808 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:13.450162 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.450931 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.452195 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.452398 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.452465 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.455481 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:13.455484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:13.455485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:13.455495 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:13.456908 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.457379 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.457381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.457381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.457446 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.457784 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.457785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.457785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.457918 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:13.457975 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:13.458295 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 02:37:13.458378 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:37:13.458686 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.447653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34307 (local address 127.2.251.3:40935)
0811 02:37:13.447914 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:13.447963 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:13.448523 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:13.448884 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:13.448888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:13.449645 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:13.450016 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:13.450025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.451106 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.451107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:13.452079 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:13.452082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.452477 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.452477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:13.452593 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.453152 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:13.453162 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:13.455350 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:13.457001 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:13.457010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:13.457013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:13.457283 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:13.457541 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:13.457542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:13.457542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:13.457681 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:13.457977 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:13.457983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:13.458127 (+ 144us) client_negotiation.cc:769] Sending connection context
0811 02:37:13.458214 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 02:37:13.458379 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":147,"threads_started":1}
I20260811 02:37:13.459534 3829 heartbeater.cc:347] Connected to a master server at 127.2.251.62:34307
I20260811 02:37:13.459645 3829 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:13.459852 3829 heartbeater.cc:511] Master 127.2.251.62:34307 requested a full tablet report, sending...
I20260811 02:37:13.460289 3132 ts_manager.cc:194] Registered new tserver with Master: 9fe395441d7a4e619e7bed21aef20d7f (127.2.251.3:40333)
I20260811 02:37:13.460655 3132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:40935
I20260811 02:37:13.462253 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:13.490828 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.481008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.2.251.1:35253)
0811 02:37:13.481175 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:13.481180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:13.481191 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:13.481222 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:13.481224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:13.481238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:13.481322 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:13.482025 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.482793 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.483923 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.484282 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.484353 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.487350 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:13.487363 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:13.487364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:13.487373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:13.488998 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.489668 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.489672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.489672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.489781 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.490024 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.490025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.490025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.490226 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:13.490357 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:13.490636 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 02:37:13.490735 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260811 02:37:13.491369 3295 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36996
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:13.518218 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:13.509809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.2.251.1:35253)
0811 02:37:13.509872 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:13.509874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:13.509984 (+ 110us) server_negotiation.cc:407] Connection header received
0811 02:37:13.510027 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:13.510028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:13.510043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:13.510113 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:13.510762 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.511281 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.512364 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:13.512600 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:13.512665 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:13.515269 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:13.515271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:13.515271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:13.515280 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:13.516661 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.517312 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.517313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.517313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.517381 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:13.517591 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:13.517592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:13.517593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:13.517709 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:13.517881 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:13.518052 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 02:37:13.518134 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:37:13.518950 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3230
I20260811 02:37:13.545171 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3425
I20260811 02:37:13.595897 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3633
I20260811 02:37:13.608565 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3083
2026-08-11T02:37:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2308 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3850](info): set up 2 sockets
Aug 11 02:37:13 dist-test-slave-cvwc krb5kdc[3850](info): commencing operation
krb5kdc: starting...
W20260811 02:37:14.823997 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:14 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:14.958604 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:33523
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:15.045945 3868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:15.046185 3868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:15.046221 3868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:15.047498 3868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:15.047546 3868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:15.047565 3868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:15.047581 3868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:15.047597 3868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:15.049234 3868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:33523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:15.049516 3868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:15.049784 3868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:15.054925 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.055024 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.054934 3883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:15.055259 3868 server_base.cc:1034] running on GCE node
I20260811 02:37:15.055584 3868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:15.055886 3868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:15.057080 3868 hybrid_clock.cc:648] HybridClock initialized: now 1786415835057047 us; error 42 us; skew 500 ppm
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:15.060544 3868 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:15.061031 3868 webserver.cc:460] Webserver started at http://127.2.251.62:41651/ using document root <none> and password file <none>
I20260811 02:37:15.061282 3868 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:15.061342 3868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:15.061474 3868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:15.062620 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "8bd61501355d44099c53d8b3bc708af9"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "e03d3f1059bc0079d868f9f45f846fef"
server_key_iv: "260b3949604baa967bd93718e445c51b"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.062965 3868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "8bd61501355d44099c53d8b3bc708af9"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "e03d3f1059bc0079d868f9f45f846fef"
server_key_iv: "260b3949604baa967bd93718e445c51b"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.064318 3868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:37:15.065248 3890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.065438 3868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:15.065531 3868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "8bd61501355d44099c53d8b3bc708af9"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "e03d3f1059bc0079d868f9f45f846fef"
server_key_iv: "260b3949604baa967bd93718e445c51b"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.065596 3868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:15.072523 3868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:15.074677 3868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:15.074847 3868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:15.078783 3868 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:33523
I20260811 02:37:15.078830 3965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:33523 every 8 connection(s)
I20260811 02:37:15.079162 3868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:15.080236 3967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:15.082661 3967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9: Bootstrap starting.
I20260811 02:37:15.083271 3967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:15.083544 3967 log.cc:824] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:15.084460 3967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9: No bootstrap required, opened a new log
I20260811 02:37:15.084965 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3868
I20260811 02:37:15.085053 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:15.085209 3052 external_mini_cluster.cc:1428] Setting key ca17153a73962a53f242d3de75ae45c5
I20260811 02:37:15.086629 3967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd61501355d44099c53d8b3bc708af9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33523 } }
I20260811 02:37:15.086818 3967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:15.086856 3967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd61501355d44099c53d8b3bc708af9, State: Initialized, Role: FOLLOWER
I20260811 02:37:15.086987 3967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd61501355d44099c53d8b3bc708af9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33523 } }
I20260811 02:37:15.087064 3967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:15.087086 3967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:15.087121 3967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:15.088393 3967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd61501355d44099c53d8b3bc708af9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33523 } }
I20260811 02:37:15.088515 3967 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd61501355d44099c53d8b3bc708af9; no voters:
I20260811 02:37:15.088692 3967 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:15.088780 3973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:15.088932 3973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd61501355d44099c53d8b3bc708af9, State: Running, Role: LEADER
I20260811 02:37:15.088953 3967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:15.089031 3973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd61501355d44099c53d8b3bc708af9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33523 } }
I20260811 02:37:15.089399 3975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd61501355d44099c53d8b3bc708af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd61501355d44099c53d8b3bc708af9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33523 } } }
I20260811 02:37:15.089387 3973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd61501355d44099c53d8b3bc708af9. Latest consensus state: current_term: 1 leader_uuid: "8bd61501355d44099c53d8b3bc708af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd61501355d44099c53d8b3bc708af9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33523 } } }
I20260811 02:37:15.089490 3975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:15.089497 3973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:15.089798 3982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:15.090350 3982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:15.092983 3982 catalog_manager.cc:1347] Generated new cluster ID: 2ad18299739149cc87bdfdd16dcf0d13
I20260811 02:37:15.093039 3982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:15.097149 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.085798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.2.251.62:33523)
0811 02:37:15.086027 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.086037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.086074 (+ 37us) server_negotiation.cc:407] Connection header received
0811 02:37:15.086589 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.086600 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.086667 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.086822 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.087548 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.088957 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.090225 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.090872 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.091329 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.093431 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.093436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.093436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.093449 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.095109 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.095465 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.095468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.095468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.095575 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.095760 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.095761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.095761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.096012 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.096805 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.096870 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.096991 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260811 02:37:15.129161 3982 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:15.129956 3982 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:15.134181 3982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd61501355d44099c53d8b3bc708af9: Generated new TSK 0
I20260811 02:37:15.134390 3982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:15.170250 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33523
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:15.258992 4002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:15.259222 4002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:15.259248 4002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:15.260437 4002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:15.260478 4002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:15.260506 4002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:15.262104 4002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:15.262312 4002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:15.262512 4002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:15.267777 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.267777 4014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.267864 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:15.268028 4002 server_base.cc:1034] running on GCE node
I20260811 02:37:15.268311 4002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:15.268570 4002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:15.269766 4002 hybrid_clock.cc:648] HybridClock initialized: now 1786415835269761 us; error 43 us; skew 500 ppm
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:15.272781 4002 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:15.273270 4002 webserver.cc:460] Webserver started at http://127.2.251.1:37419/ using document root <none> and password file <none>
I20260811 02:37:15.273492 4002 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:15.273547 4002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:15.273687 4002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:15.274864 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "ce5a22aa16314778bea536c222b052e6"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "c4acc63e9893bcdd130ce783d08eecc4"
server_key_iv: "513c3d22c8128ca6d9f2a38df8a410a8"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.275252 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "ce5a22aa16314778bea536c222b052e6"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "c4acc63e9893bcdd130ce783d08eecc4"
server_key_iv: "513c3d22c8128ca6d9f2a38df8a410a8"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.276777 4002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:15.277662 4022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.277881 4002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:15.277979 4002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "ce5a22aa16314778bea536c222b052e6"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "c4acc63e9893bcdd130ce783d08eecc4"
server_key_iv: "513c3d22c8128ca6d9f2a38df8a410a8"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.278064 4002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:15.311504 4002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:15.315043 4002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:15.315231 4002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:15.315650 4002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:15.316052 4002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:15.316095 4002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.316131 4002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:15.316156 4002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.323087 4002 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:38981
I20260811 02:37:15.323194 4194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:38981 every 8 connection(s)
I20260811 02:37:15.323540 4002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:15.326833 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4002
I20260811 02:37:15.326961 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:15.327138 3052 external_mini_cluster.cc:1428] Setting key ee86ec14b2b996f73926cda9faa4c6ee
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:15.335903 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.324475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:44449 (local address 127.2.251.62:33523)
0811 02:37:15.324575 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.324577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.324761 (+ 184us) server_negotiation.cc:407] Connection header received
0811 02:37:15.326146 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.326148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.326167 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.326220 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.327008 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.327606 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.328737 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.328944 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.329005 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.332149 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.332152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.332152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.332168 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.334091 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.334653 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.334654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.334655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.334720 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.335048 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.335049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.335049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.335192 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.335262 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.335708 (+ 446us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.335804 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 02:37:15.336090 4200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.324380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33523 (local address 127.2.251.1:44449)
0811 02:37:15.324609 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.324652 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.326015 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.326309 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.326313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.326653 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:15.326899 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.326905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.327700 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.327701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.328625 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.328629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.329025 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.329026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.329400 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.330072 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:15.330085 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:15.331943 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:15.334195 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:15.334206 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:15.334209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:15.334560 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:15.334786 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:15.334787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:15.334787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:15.334946 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:15.335269 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:15.335277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:15.335484 (+ 207us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.335635 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.335835 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:37:15.336686 4195 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33523
I20260811 02:37:15.336795 4195 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:15.337013 4195 heartbeater.cc:511] Master 127.2.251.62:33523 requested a full tablet report, sending...
I20260811 02:37:15.337518 3915 ts_manager.cc:194] Registered new tserver with Master: ce5a22aa16314778bea536c222b052e6 (127.2.251.1:38981)
I20260811 02:37:15.338320 3915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:44449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:15.357174 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33523
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:15.447919 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:15.448269 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:15.448304 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:15.450517 4209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:15.450616 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:15.450657 4209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:15.452574 4209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:15.452860 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:15.453104 4209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:15.458454 4221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.458408 4222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:15.458563 4209 server_base.cc:1034] running on GCE node
W20260811 02:37:15.458415 4225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:15.458875 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:15.459157 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:15.460341 4209 hybrid_clock.cc:648] HybridClock initialized: now 1786415835460308 us; error 54 us; skew 500 ppm
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:15.463107 4209 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:15.463750 4209 webserver.cc:460] Webserver started at http://127.2.251.2:42493/ using document root <none> and password file <none>
I20260811 02:37:15.463963 4209 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:15.464010 4209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:15.464107 4209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:15.465222 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "19f1f99d93cf4722a870ae62d35830bd"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "3bcdb8d4ac51ca197889be08ccc5b0a4"
server_key_iv: "b35c9be01bba989c1d8456e68dc19283"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.465581 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "19f1f99d93cf4722a870ae62d35830bd"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "3bcdb8d4ac51ca197889be08ccc5b0a4"
server_key_iv: "b35c9be01bba989c1d8456e68dc19283"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.467209 4209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:15.468569 4236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.468931 4209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:15.469043 4209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "19f1f99d93cf4722a870ae62d35830bd"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "3bcdb8d4ac51ca197889be08ccc5b0a4"
server_key_iv: "b35c9be01bba989c1d8456e68dc19283"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.469133 4209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:15.496603 4209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:15.499202 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:15.499346 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:15.499630 4209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:15.499995 4209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:15.500033 4209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.500056 4209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:15.500067 4209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.508074 4209 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:45555
I20260811 02:37:15.508216 4411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:45555 every 8 connection(s)
I20260811 02:37:15.508867 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:15.514166 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4209
I20260811 02:37:15.514339 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:15.514554 3052 external_mini_cluster.cc:1428] Setting key 11e792fe867be03352a39422e6ef9a8e
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:15.520327 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.509942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:59439 (local address 127.2.251.62:33523)
0811 02:37:15.510028 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.510031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.510192 (+ 161us) server_negotiation.cc:407] Connection header received
0811 02:37:15.511214 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.511215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.511228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.511290 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.512140 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.512651 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.513954 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.514160 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.514226 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.517091 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.517093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.517093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.517103 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.518706 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.519147 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.519148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.519149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.519213 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.519507 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.519508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.519509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.519666 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.519741 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.520125 (+ 384us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.520221 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 02:37:15.520435 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.509820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33523 (local address 127.2.251.2:59439)
0811 02:37:15.510033 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.510088 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.511081 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.511429 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.511433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.511728 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:15.511944 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.511949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.512763 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.512765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.513780 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.513784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.514330 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.514335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.514520 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.515266 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:15.515277 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:15.516971 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:15.518811 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:15.518819 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:15.518822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:15.519067 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:15.519308 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:15.519310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:15.519310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:15.519431 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:15.519762 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:15.519770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:15.519978 (+ 208us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.520087 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.520238 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260811 02:37:15.521028 4415 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33523
I20260811 02:37:15.521143 4415 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:15.521341 4415 heartbeater.cc:511] Master 127.2.251.62:33523 requested a full tablet report, sending...
I20260811 02:37:15.521826 3915 ts_manager.cc:194] Registered new tserver with Master: 19f1f99d93cf4722a870ae62d35830bd (127.2.251.2:45555)
I20260811 02:37:15.522339 3915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:59439
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:15.547145 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33523
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:15.643707 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:15.644027 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:15.644061 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:15.645569 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:15.645635 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:15.645669 4425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:15.647842 4425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33523
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:15.648223 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:15.648526 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:15.654569 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.654570 4433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:15.654569 4436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:15.654902 4425 server_base.cc:1034] running on GCE node
I20260811 02:37:15.655694 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:15.656028 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:15.657228 4425 hybrid_clock.cc:648] HybridClock initialized: now 1786415835657214 us; error 35 us; skew 500 ppm
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:15.660063 4425 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:15.660589 4425 webserver.cc:460] Webserver started at http://127.2.251.3:40071/ using document root <none> and password file <none>
I20260811 02:37:15.660806 4425 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:15.660853 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:15.660980 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:15.662166 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "08528dfe7680497b9c0004f5da510ea6"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "0513df67c546abec27bd558180300964"
server_key_iv: "a1b51d5a0e5f03939ecfb4507b3cd5f5"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.662617 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "08528dfe7680497b9c0004f5da510ea6"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "0513df67c546abec27bd558180300964"
server_key_iv: "a1b51d5a0e5f03939ecfb4507b3cd5f5"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.664197 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:15.665136 4444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.665467 4425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:15.665673 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "08528dfe7680497b9c0004f5da510ea6"
format_stamp: "Formatted at 2026-08-11 02:37:15 on dist-test-slave-cvwc"
server_key: "0513df67c546abec27bd558180300964"
server_key_iv: "a1b51d5a0e5f03939ecfb4507b3cd5f5"
server_key_version: "encryptionkey@0"
I20260811 02:37:15.665832 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:15.679044 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:15.681615 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:15.681790 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:15.682044 4425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:15.682366 4425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:15.682406 4425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.682442 4425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:15.682466 4425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:15.688488 4425 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:44399
I20260811 02:37:15.688563 4620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:44399 every 8 connection(s)
I20260811 02:37:15.688903 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:15.694821 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4425
I20260811 02:37:15.694922 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:15.695073 3052 external_mini_cluster.cc:1428] Setting key 2f39f54def6c81c60d977fabaa1a234e
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:15.700282 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.689823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:40991 (local address 127.2.251.62:33523)
0811 02:37:15.689871 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.689874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.690098 (+ 224us) server_negotiation.cc:407] Connection header received
0811 02:37:15.691410 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.691412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.691428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.691507 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.692175 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.692685 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.693765 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.693991 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.694059 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.697426 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.697428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.697428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.697437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.698908 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.699337 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.699338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.699338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.699393 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.699636 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.699636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.699637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.699754 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.699810 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.700104 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.700189 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:15.700491 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.689780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33523 (local address 127.2.251.3:40991)
0811 02:37:15.689968 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.690010 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.691239 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.691576 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.691579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.691868 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:15.692067 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.692071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.692794 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.692795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.693640 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.693643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.694176 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.694179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.694393 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.695309 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:15.695323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:15.697317 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:15.699006 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:15.699016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:15.699019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:15.699265 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:15.699461 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:15.699462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:15.699462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:15.699578 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:15.699809 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:15.699817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:15.699980 (+ 163us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.700073 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.700254 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260811 02:37:15.701174 4629 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33523
I20260811 02:37:15.701277 4629 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:15.701479 4629 heartbeater.cc:511] Master 127.2.251.62:33523 requested a full tablet report, sending...
I20260811 02:37:15.701969 3915 ts_manager.cc:194] Registered new tserver with Master: 08528dfe7680497b9c0004f5da510ea6 (127.2.251.3:44399)
I20260811 02:37:15.702355 3915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:40991
I20260811 02:37:15.707594 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:37:15.716939 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.710172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.2.251.62:33523)
0811 02:37:15.710333 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.710335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.710440 (+ 105us) server_negotiation.cc:407] Connection header received
0811 02:37:15.710524 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.710525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.710537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.710586 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.711183 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.711686 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.712383 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.712567 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.712622 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.713981 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.713982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.713983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.713989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.715629 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.715982 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.715983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.715983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.716038 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.716252 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.716253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.716253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.716508 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.716622 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.716749 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.716842 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 02:37:15.720189 3915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:15.722002 3915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:15.729802 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.725507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44399 (local address 127.0.0.1:46996)
0811 02:37:15.725845 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.725856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.725914 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.726303 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.726304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.726306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:15.726361 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.726363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.727350 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.727351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.728657 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.728659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.729558 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.729559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.729615 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.729618 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.729660 (+ 42us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.729691 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260811 02:37:15.729842 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.725899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.2.251.3:44399)
0811 02:37:15.726061 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.726066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.726078 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:15.726119 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.726121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.726138 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.726216 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:15.726439 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.727243 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.728744 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.729485 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.729559 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.729617 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.729687 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.729757 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260811 02:37:15.729803 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.725669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:45555 (local address 127.0.0.1:50818)
0811 02:37:15.726257 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.726268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.726323 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.726661 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.726662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.726664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:15.726702 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.726704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.727814 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.727815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.728619 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.728622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.729422 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.729423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.729503 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.729519 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.729587 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.729644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":106,"threads_started":1}
I20260811 02:37:15.729842 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.726079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.2.251.2:45555)
0811 02:37:15.726263 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.726267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.726314 (+ 47us) server_negotiation.cc:407] Connection header received
0811 02:37:15.726383 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.726385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.726399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.726559 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:15.726780 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.727725 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.728733 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.729317 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.729405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.729472 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.729685 (+ 213us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.729753 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260811 02:37:15.730253 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.725456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:38981 (local address 127.0.0.1:51006)
0811 02:37:15.725672 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.725734 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.725817 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.726065 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.726069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.726071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:15.726146 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.726148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.727547 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.727548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.728641 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.728644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.730020 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.730021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.730099 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.730104 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.730154 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.730196 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":11,"thread_start_us":72,"threads_started":1}
I20260811 02:37:15.730521 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.725683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.2.251.1:38981)
0811 02:37:15.725891 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.725896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.725908 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:15.725952 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.725954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.725970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.726046 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:15.726231 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.727420 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.728745 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.729488 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.729559 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.729621 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.730392 (+ 771us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.730444 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20260811 02:37:15.730919 4316 tablet_service.cc:1467] Processing CreateTablet for tablet 27a9380892a84bc0acfe1e9f946f283c (DEFAULT_TABLE table=test-table [id=def26e9a77f94fffbb5047c0e8c43b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:15.731034 4097 tablet_service.cc:1467] Processing CreateTablet for tablet 27a9380892a84bc0acfe1e9f946f283c (DEFAULT_TABLE table=test-table [id=def26e9a77f94fffbb5047c0e8c43b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:15.731321 4097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a9380892a84bc0acfe1e9f946f283c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:15.731323 4316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a9380892a84bc0acfe1e9f946f283c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:15.733686 4524 tablet_service.cc:1467] Processing CreateTablet for tablet 27a9380892a84bc0acfe1e9f946f283c (DEFAULT_TABLE table=test-table [id=def26e9a77f94fffbb5047c0e8c43b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:15.734073 4524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a9380892a84bc0acfe1e9f946f283c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:15.734628 4665 tablet_bootstrap.cc:492] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: Bootstrap starting.
I20260811 02:37:15.735240 4665 tablet_bootstrap.cc:654] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:15.735811 4665 log.cc:824] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:15.736205 4667 tablet_bootstrap.cc:492] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: Bootstrap starting.
I20260811 02:37:15.736665 4665 tablet_bootstrap.cc:492] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: No bootstrap required, opened a new log
I20260811 02:37:15.736742 4665 ts_tablet_manager.cc:1397] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:15.736944 4667 tablet_bootstrap.cc:654] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:15.737257 4673 tablet_bootstrap.cc:492] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: Bootstrap starting.
I20260811 02:37:15.737705 4667 log.cc:824] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:15.738256 4673 tablet_bootstrap.cc:654] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:15.738679 4667 tablet_bootstrap.cc:492] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: No bootstrap required, opened a new log
I20260811 02:37:15.738770 4667 ts_tablet_manager.cc:1397] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:37:15.738816 4673 log.cc:824] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:15.738837 4665 raft_consensus.cc:348] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.739043 4665 raft_consensus.cc:374] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:15.739091 4665 raft_consensus.cc:729] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f1f99d93cf4722a870ae62d35830bd, State: Initialized, Role: FOLLOWER
I20260811 02:37:15.739216 4665 consensus_queue.cc:260] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.739519 4665 ts_tablet_manager.cc:1428] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:15.739671 4415 heartbeater.cc:503] Master 127.2.251.62:33523 was elected leader, sending a full tablet report...
I20260811 02:37:15.739861 4673 tablet_bootstrap.cc:492] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: No bootstrap required, opened a new log
I20260811 02:37:15.739943 4673 ts_tablet_manager.cc:1397] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:37:15.740739 4667 raft_consensus.cc:348] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.740893 4667 raft_consensus.cc:374] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:15.740940 4667 raft_consensus.cc:729] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5a22aa16314778bea536c222b052e6, State: Initialized, Role: FOLLOWER
I20260811 02:37:15.741063 4667 consensus_queue.cc:260] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.741351 4667 ts_tablet_manager.cc:1428] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:15.741407 4195 heartbeater.cc:503] Master 127.2.251.62:33523 was elected leader, sending a full tablet report...
I20260811 02:37:15.742203 4673 raft_consensus.cc:348] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.742389 4673 raft_consensus.cc:374] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:15.742437 4673 raft_consensus.cc:729] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08528dfe7680497b9c0004f5da510ea6, State: Initialized, Role: FOLLOWER
I20260811 02:37:15.742558 4673 consensus_queue.cc:260] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.742852 4673 ts_tablet_manager.cc:1428] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:15.742898 4629 heartbeater.cc:503] Master 127.2.251.62:33523 was elected leader, sending a full tablet report...
W20260811 02:37:15.759765 4416 tablet.cc:2369] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:37:15.824421 4198 tablet.cc:2369] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:15.833827 4680 raft_consensus.cc:482] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:15.833983 4680 raft_consensus.cc:504] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.834554 4680 leader_election.cc:288] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce5a22aa16314778bea536c222b052e6 (127.2.251.1:38981), 08528dfe7680497b9c0004f5da510ea6 (127.2.251.3:44399)
I20260811 02:37:15.837867 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.834999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:50557 (local address 127.2.251.1:38981)
0811 02:37:15.835065 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.835067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.835111 (+ 44us) server_negotiation.cc:407] Connection header received
0811 02:37:15.835194 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.835195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.835202 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.835249 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:15.835473 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.836050 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.837115 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.837485 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.837549 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.837562 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.837745 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.837789 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 02:37:15.837872 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.835001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:50015 (local address 127.2.251.3:44399)
0811 02:37:15.835080 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.835083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.835093 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:15.835124 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.835125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.835139 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.835187 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:15.835360 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.835938 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.837037 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.837461 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.837538 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.837553 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.837740 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.837789 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 02:37:15.837898 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.834825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44399 (local address 127.2.251.2:50015)
0811 02:37:15.834927 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.834943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.835033 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.835202 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.835204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.835205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:15.835287 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.835289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.836073 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.836074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.836923 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.836927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.837556 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.837557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.837639 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.837642 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.837699 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.837793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 02:37:15.837916 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.834846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:38981 (local address 127.2.251.2:50557)
0811 02:37:15.835021 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:15.835031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:15.835120 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:15.835258 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:15.835259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:15.835261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:15.835339 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.835341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.836163 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.836164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.837019 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:15.837022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.837573 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.837574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:15.837644 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.837647 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:15.837694 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:15.837768 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260811 02:37:15.838411 4130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a9380892a84bc0acfe1e9f946f283c" candidate_uuid: "19f1f99d93cf4722a870ae62d35830bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5a22aa16314778bea536c222b052e6" is_pre_election: true
I20260811 02:37:15.838634 4130 raft_consensus.cc:2454] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f1f99d93cf4722a870ae62d35830bd in term 0.
I20260811 02:37:15.838948 4244 leader_election.cc:302] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f1f99d93cf4722a870ae62d35830bd, ce5a22aa16314778bea536c222b052e6; no voters:
I20260811 02:37:15.839124 4680 raft_consensus.cc:2781] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:15.839223 4680 raft_consensus.cc:482] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:15.839262 4680 raft_consensus.cc:3037] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:15.840114 4680 raft_consensus.cc:504] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.840296 4680 leader_election.cc:288] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [CANDIDATE]: Term 1 election: Requested vote from peers ce5a22aa16314778bea536c222b052e6 (127.2.251.1:38981), 08528dfe7680497b9c0004f5da510ea6 (127.2.251.3:44399)
I20260811 02:37:15.840054 4553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a9380892a84bc0acfe1e9f946f283c" candidate_uuid: "19f1f99d93cf4722a870ae62d35830bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08528dfe7680497b9c0004f5da510ea6" is_pre_election: true
I20260811 02:37:15.840456 4553 raft_consensus.cc:2454] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f1f99d93cf4722a870ae62d35830bd in term 0.
I20260811 02:37:15.840536 4130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a9380892a84bc0acfe1e9f946f283c" candidate_uuid: "19f1f99d93cf4722a870ae62d35830bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5a22aa16314778bea536c222b052e6"
I20260811 02:37:15.840600 4551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a9380892a84bc0acfe1e9f946f283c" candidate_uuid: "19f1f99d93cf4722a870ae62d35830bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08528dfe7680497b9c0004f5da510ea6"
I20260811 02:37:15.840654 4130 raft_consensus.cc:3037] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:15.840688 4551 raft_consensus.cc:3037] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:15.841518 4130 raft_consensus.cc:2454] T 27a9380892a84bc0acfe1e9f946f283c P ce5a22aa16314778bea536c222b052e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f1f99d93cf4722a870ae62d35830bd in term 1.
I20260811 02:37:15.841781 4244 leader_election.cc:302] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f1f99d93cf4722a870ae62d35830bd, ce5a22aa16314778bea536c222b052e6; no voters:
I20260811 02:37:15.841801 4551 raft_consensus.cc:2454] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f1f99d93cf4722a870ae62d35830bd in term 1.
I20260811 02:37:15.841930 4680 raft_consensus.cc:2781] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:15.842120 4680 raft_consensus.cc:686] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [term 1 LEADER]: Becoming Leader. State: Replica: 19f1f99d93cf4722a870ae62d35830bd, State: Running, Role: LEADER
I20260811 02:37:15.842216 4680 consensus_queue.cc:237] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } }
I20260811 02:37:15.843334 3914 catalog_manager.cc:5495] T 27a9380892a84bc0acfe1e9f946f283c P 19f1f99d93cf4722a870ae62d35830bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f1f99d93cf4722a870ae62d35830bd (127.2.251.2). New cstate: current_term: 1 leader_uuid: "19f1f99d93cf4722a870ae62d35830bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a22aa16314778bea536c222b052e6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 38981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08528dfe7680497b9c0004f5da510ea6" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19f1f99d93cf4722a870ae62d35830bd" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45555 } health_report { overall_health: HEALTHY } } }
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:15.886269 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.874430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.2.251.1:38981)
0811 02:37:15.874491 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.874493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.874719 (+ 226us) server_negotiation.cc:407] Connection header received
0811 02:37:15.874777 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.874778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.874791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.874847 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.875628 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.876434 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.877523 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.877940 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.878005 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.882124 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.882146 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.882147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.882158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.884392 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.884918 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.884924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.884924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.885023 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.885307 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.885308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.885309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.885548 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.885752 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.886026 (+ 274us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.886142 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:15.920102 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.912552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.2.251.1:38981)
0811 02:37:15.912617 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.912619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.912676 (+ 57us) server_negotiation.cc:407] Connection header received
0811 02:37:15.912744 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.912745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.912755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.912811 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.913442 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.913987 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.914733 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.914924 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.914981 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.917305 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.917308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.917308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.917319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.918970 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.919365 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.919367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.919367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.919435 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.919651 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.919651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.919652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.919748 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.919828 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.919958 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.920028 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260811 02:37:15.939582 4631 tablet.cc:2369] T 27a9380892a84bc0acfe1e9f946f283c P 08528dfe7680497b9c0004f5da510ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 02:37:15 dist-test-slave-cvwc krb5kdc[3850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:15.945914 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:15.938489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.2.251.1:38981)
0811 02:37:15.938585 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:15.938587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:15.938598 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:15.938631 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:15.938632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:15.938642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:15.938705 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:15.939316 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.939844 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.940890 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:15.941115 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:15.941173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:15.943408 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:15.943410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:15.943410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:15.943418 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:15.944859 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.945191 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.945192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.945192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.945246 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:15.945421 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:15.945422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:15.945422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:15.945527 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:15.945614 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:15.945731 (+ 117us) server_negotiation.cc:299] Negotiation successful
0811 02:37:15.945811 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 02:37:15.946614 4060 tablet_service.cc:2908] Rejecting scan request for tablet 27a9380892a84bc0acfe1e9f946f283c: Uninitialized: clean time has not yet been initialized
I20260811 02:37:15.948395 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4002
I20260811 02:37:15.989878 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4209
I20260811 02:37:16.000301 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4425
I20260811 02:37:16.011900 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3868
2026-08-11T02:37:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2400 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:16 dist-test-slave-cvwc krb5kdc[4707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:16 dist-test-slave-cvwc krb5kdc[4707](info): set up 2 sockets
Aug 11 02:37:16 dist-test-slave-cvwc krb5kdc[4707](info): commencing operation
krb5kdc: starting...
W20260811 02:37:17.254210 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:17.393879 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34429
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:17.483901 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:17.484153 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:17.484179 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:17.485450 4730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:17.485517 4730 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 02:37:17.485532 4730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:17.485546 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:17.485560 4730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:17.485574 4730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:17.487280 4730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34429
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:17.487545 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:17.487775 4730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:17.493230 4741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:17.493207 4738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:17.493242 4730 server_base.cc:1034] running on GCE node
W20260811 02:37:17.493209 4739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:17.493844 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:17.494170 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:17.495533 4730 hybrid_clock.cc:648] HybridClock initialized: now 1786415837495496 us; error 50 us; skew 500 ppm
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:17.505570 4730 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:17.506242 4730 webserver.cc:460] Webserver started at http://127.2.251.62:37093/ using document root <none> and password file <none>
I20260811 02:37:17.506510 4730 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:17.506561 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:17.506669 4730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:17.507831 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "d0d963c7183b41fca76c3881c5490302"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "767863c3aa0c9c3382df75f7342d3b54"
server_key_iv: "9a53758f67680fad6dcff0382924e4e1"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.508204 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "d0d963c7183b41fca76c3881c5490302"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "767863c3aa0c9c3382df75f7342d3b54"
server_key_iv: "9a53758f67680fad6dcff0382924e4e1"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.510058 4730 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 02:37:17.511171 4753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.511441 4730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:17.511621 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "d0d963c7183b41fca76c3881c5490302"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "767863c3aa0c9c3382df75f7342d3b54"
server_key_iv: "9a53758f67680fad6dcff0382924e4e1"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.511739 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:17.522888 4730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:17.525425 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:17.525579 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:17.531294 4730 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:34429
I20260811 02:37:17.531361 4851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:34429 every 8 connection(s)
I20260811 02:37:17.531737 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:17.532565 4856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:17.535045 4856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: Bootstrap starting.
I20260811 02:37:17.535818 4856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:17.536288 4856 log.cc:824] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:17.537384 4856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: No bootstrap required, opened a new log
I20260811 02:37:17.541143 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4730
I20260811 02:37:17.541235 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:17.541074 4856 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d963c7183b41fca76c3881c5490302" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34429 } }
I20260811 02:37:17.541283 4856 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:17.541323 4856 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d963c7183b41fca76c3881c5490302, State: Initialized, Role: FOLLOWER
I20260811 02:37:17.541396 3052 external_mini_cluster.cc:1428] Setting key 5c5249e98026b619a8f55fdd1e07117e
I20260811 02:37:17.541920 4856 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d963c7183b41fca76c3881c5490302" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34429 } }
I20260811 02:37:17.542240 4856 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:17.542403 4856 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:17.542496 4856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:17.543288 4856 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d963c7183b41fca76c3881c5490302" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34429 } }
I20260811 02:37:17.543530 4856 leader_election.cc:302] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d963c7183b41fca76c3881c5490302; no voters:
I20260811 02:37:17.543756 4856 leader_election.cc:288] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:17.543923 4861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:17.544085 4861 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [term 1 LEADER]: Becoming Leader. State: Replica: d0d963c7183b41fca76c3881c5490302, State: Running, Role: LEADER
I20260811 02:37:17.544257 4861 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d963c7183b41fca76c3881c5490302" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34429 } }
I20260811 02:37:17.544396 4856 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:17.544684 4861 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d963c7183b41fca76c3881c5490302. Latest consensus state: current_term: 1 leader_uuid: "d0d963c7183b41fca76c3881c5490302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d963c7183b41fca76c3881c5490302" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34429 } } }
I20260811 02:37:17.544962 4861 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:17.544888 4862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d963c7183b41fca76c3881c5490302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d963c7183b41fca76c3881c5490302" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34429 } } }
I20260811 02:37:17.545401 4862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302 [sys.catalog]: This master's current role is: LEADER
W20260811 02:37:17.546276 4883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:37:17.546525 4883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:37:17.546603 4875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:17.546795 4875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:17.548672 4875 catalog_manager.cc:1347] Generated new cluster ID: 1cfd84fcdbf94d7c854c42eeba4ae1ca
I20260811 02:37:17.548744 4875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:17.553903 4875 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:17.554620 4875 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:17.555266 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.542018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.2.251.62:34429)
0811 02:37:17.542366 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:17.542378 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:17.542413 (+ 35us) server_negotiation.cc:407] Connection header received
0811 02:37:17.542556 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:17.542564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:17.542624 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:17.542757 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:17.543605 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.545456 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.546378 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.547027 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.547152 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.550261 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:17.550266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:17.550267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:17.550281 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:17.553070 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.553568 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.553572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.553572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.553695 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.553973 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.553975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.553975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.554265 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:17.554440 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:17.554898 (+ 458us) server_negotiation.cc:299] Negotiation successful
0811 02:37:17.555051 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":244,"threads_started":1}
I20260811 02:37:17.563287 4875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: Generated new TSK 0
I20260811 02:37:17.563498 4875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:17.597645 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34429
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:17.690228 4888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:17.690441 4888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:17.690466 4888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:17.691640 4888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:17.691685 4888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:17.691706 4888 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:17.691723 4888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:17.693207 4888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:17.693447 4888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:17.693682 4888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:17.699085 4906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:17.699085 4902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:17.699077 4904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:17.699546 4888 server_base.cc:1034] running on GCE node
I20260811 02:37:17.699879 4888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:17.700218 4888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:17.701399 4888 hybrid_clock.cc:648] HybridClock initialized: now 1786415837701349 us; error 70 us; skew 500 ppm
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:17.704556 4888 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:17.705046 4888 webserver.cc:460] Webserver started at http://127.2.251.1:40973/ using document root <none> and password file <none>
I20260811 02:37:17.705243 4888 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:17.705283 4888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:17.705387 4888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:17.706504 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "eb636642614e449799bd41049493772f"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "892b522c33f18369caada54601c63c8a"
server_key_iv: "9d307a3512612343be32756cf8200e71"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.706831 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "eb636642614e449799bd41049493772f"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "892b522c33f18369caada54601c63c8a"
server_key_iv: "9d307a3512612343be32756cf8200e71"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.708194 4888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:17.709129 4919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.709342 4888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:17.709434 4888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "eb636642614e449799bd41049493772f"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "892b522c33f18369caada54601c63c8a"
server_key_iv: "9d307a3512612343be32756cf8200e71"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.709501 4888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:17.722566 4888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:17.725225 4888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:17.725365 4888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:17.726023 4888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:17.726068 4888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.726091 4888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:17.726102 4888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.732887 4888 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:45681
I20260811 02:37:17.733040 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:45681 every 8 connection(s)
I20260811 02:37:17.733343 4888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:17.734243 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4888
I20260811 02:37:17.734328 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:17.734488 3052 external_mini_cluster.cc:1428] Setting key a301780619dba943e0878f6c2bec16a0
I20260811 02:37:17.737924 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.726633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:34941 (local address 127.2.251.62:34429)
0811 02:37:17.726713 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:17.726716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:17.726894 (+ 178us) server_negotiation.cc:407] Connection header received
0811 02:37:17.727461 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:17.727462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:17.727474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:17.727556 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:17.728528 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.729310 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.730920 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.731237 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.731318 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.734847 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:17.734849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:17.734849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:17.734858 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:17.736414 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.736869 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.736871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.736871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.736937 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.737190 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.737191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.737192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.737327 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:17.737392 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:17.737682 (+ 290us) server_negotiation.cc:299] Negotiation successful
0811 02:37:17.737830 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:37:17.738102 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.726569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.2.251.1:34941)
0811 02:37:17.726755 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:17.726799 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:17.727345 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:17.727643 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:17.727648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:17.728023 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:17.728345 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.728353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.729427 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.729428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.730800 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.730805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.731359 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.731361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.731544 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.732502 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:17.732511 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:17.734729 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:17.736512 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.736521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.736524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.736788 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.737002 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.737003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.737003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.737126 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.737391 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:17.737399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:17.737550 (+ 151us) client_negotiation.cc:769] Sending connection context
0811 02:37:17.737650 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 02:37:17.737872 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260811 02:37:17.738651 5100 heartbeater.cc:347] Connected to a master server at 127.2.251.62:34429
I20260811 02:37:17.738759 5100 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:17.738983 5100 heartbeater.cc:511] Master 127.2.251.62:34429 requested a full tablet report, sending...
I20260811 02:37:17.739557 4777 ts_manager.cc:194] Registered new tserver with Master: eb636642614e449799bd41049493772f (127.2.251.1:45681)
I20260811 02:37:17.740383 4777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:34941
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:37:17.746702 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.740155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.2.251.1:45331)
0811 02:37:17.740328 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:17.740341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:17.740485 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:17.740714 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:17.740715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:17.740944 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:17.741023 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.741026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.741774 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.741775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.742564 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.742568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.742953 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.742954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.743037 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.743520 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:17.743525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:17.743885 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:17.745621 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.745622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.745623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.745840 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.746058 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.746059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.746060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.746128 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.746397 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:17.746399 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:17.746450 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 02:37:17.746526 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:37:17.746622 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260811 02:37:17.746686 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.740080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:45331 (local address 127.2.251.62:34429)
0811 02:37:17.740123 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:17.740124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:17.740418 (+ 294us) server_negotiation.cc:407] Connection header received
0811 02:37:17.740605 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:17.740607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:17.740619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:17.740649 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:17.741113 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.741647 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.742673 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.742868 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.742932 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.743991 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:17.743992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:17.743992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:17.743999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:17.745538 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.745914 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.745914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.745915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.745974 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.746183 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.746184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.746184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.746342 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:17.746432 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:17.746544 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 02:37:17.746618 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:17.765157 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34429
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:17.857595 5114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:17.857853 5114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:17.857901 5114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:17.859680 5114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:17.860033 5114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:17.860126 5114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:17.860165 5114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:17.862540 5114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:17.863114 5114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:17.863373 5114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:17.868716 5130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:17.868742 5126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:17.868775 5114 server_base.cc:1034] running on GCE node
W20260811 02:37:17.868737 5128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:17.869220 5114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:17.869493 5114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:17.870664 5114 hybrid_clock.cc:648] HybridClock initialized: now 1786415837870632 us; error 46 us; skew 500 ppm
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:17.873379 5114 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:17.873847 5114 webserver.cc:460] Webserver started at http://127.2.251.2:36759/ using document root <none> and password file <none>
I20260811 02:37:17.874045 5114 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:17.874085 5114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:17.874184 5114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:17.875243 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "a4b5d205ffd14374826fa57b7a8cf22b"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "49cbb18d52ffe8293b54a87a8e99f7b2"
server_key_iv: "51fdc3d1278859d7d96c534fb25df64f"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.875572 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "a4b5d205ffd14374826fa57b7a8cf22b"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "49cbb18d52ffe8293b54a87a8e99f7b2"
server_key_iv: "51fdc3d1278859d7d96c534fb25df64f"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.876959 5114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:17.877825 5141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.877978 5114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:17.878067 5114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "a4b5d205ffd14374826fa57b7a8cf22b"
format_stamp: "Formatted at 2026-08-11 02:37:17 on dist-test-slave-cvwc"
server_key: "49cbb18d52ffe8293b54a87a8e99f7b2"
server_key_iv: "51fdc3d1278859d7d96c534fb25df64f"
server_key_version: "encryptionkey@0"
I20260811 02:37:17.878136 5114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:17.886849 5114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:17.889003 5114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:17.889130 5114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:17.889746 5114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:17.889801 5114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.889827 5114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:17.889851 5114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:17.897063 5114 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:42777
I20260811 02:37:17.897140 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:42777 every 8 connection(s)
Aug 11 02:37:17 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:17.897535 5114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:17.901788 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.890160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:56649 (local address 127.2.251.62:34429)
0811 02:37:17.890228 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:17.890231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:17.890447 (+ 216us) server_negotiation.cc:407] Connection header received
0811 02:37:17.890943 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:17.890944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:17.890958 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:17.891037 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:17.891854 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.892644 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.894172 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.894485 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.894577 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.898022 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:17.898024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:17.898025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:17.898032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:17.899619 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.900207 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.900208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.900208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.900274 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.900667 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.900668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.900669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.900808 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:17.900887 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:17.901559 (+ 672us) server_negotiation.cc:299] Negotiation successful
0811 02:37:17.901658 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:37:17.902195 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5114
I20260811 02:37:17.902294 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:17.902478 3052 external_mini_cluster.cc:1428] Setting key 63e19ba778d5c203117e8250a4b3dd98
I20260811 02:37:17.902630 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.890153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.2.251.2:56649)
0811 02:37:17.890315 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:17.890356 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:17.890843 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:17.891107 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:17.891111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:17.891459 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:17.891747 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.891753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.892747 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.892748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.894065 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.894069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.894587 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.894588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.894741 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.895327 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:17.895350 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:17.897908 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:17.899759 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.899767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.899769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.900074 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.900446 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.900447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.900447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.900578 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.900946 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:17.900953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:17.901094 (+ 141us) client_negotiation.cc:769] Sending connection context
0811 02:37:17.901499 (+ 405us) client_negotiation.cc:240] Negotiation successful
0811 02:37:17.901666 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260811 02:37:17.903394 5332 heartbeater.cc:347] Connected to a master server at 127.2.251.62:34429
I20260811 02:37:17.903522 5332 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:17.903856 5332 heartbeater.cc:511] Master 127.2.251.62:34429 requested a full tablet report, sending...
I20260811 02:37:17.904582 4777 ts_manager.cc:194] Registered new tserver with Master: a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777)
I20260811 02:37:17.905063 4777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:56649
I20260811 02:37:17.911921 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.904745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.2.251.2:48659)
0811 02:37:17.905033 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:17.905055 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:17.905132 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:17.905608 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:17.905609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:17.905876 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:17.905969 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.905972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.906921 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.906922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.907636 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:17.907640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.908027 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.908028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:17.908104 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.908618 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:17.908624 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:17.909088 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:17.910642 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.910644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.910644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.910879 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.911052 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:17.911053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:17.911053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:17.911123 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:17.911486 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:17.911488 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:17.911560 (+ 72us) client_negotiation.cc:769] Sending connection context
0811 02:37:17.911672 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 02:37:17.911790 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":142,"threads_started":1}
I20260811 02:37:17.912002 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:17.905396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:48659 (local address 127.2.251.62:34429)
0811 02:37:17.905456 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:17.905458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:17.905469 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:17.905494 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:17.905495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:17.905506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:17.905573 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:17.906066 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.906761 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.907746 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:17.907947 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:17.908010 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:17.909193 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:17.909195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:17.909195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:17.909203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:17.910555 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.910947 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.910948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.910949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.910999 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:17.911226 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:17.911227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:17.911228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:17.911411 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:17.911477 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:17.911741 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 02:37:17.911858 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:17.940477 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:34429
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:18.028066 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:18.028296 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:18.028321 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:18.029528 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:18.029584 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:18.029605 5343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:18.029623 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:18.031162 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:18.031399 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:18.031629 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:18.037091 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:18.037082 5355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:18.037117 5343 server_base.cc:1034] running on GCE node
W20260811 02:37:18.037083 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:18.037503 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:18.037761 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:18.039027 5343 hybrid_clock.cc:648] HybridClock initialized: now 1786415838039019 us; error 34 us; skew 500 ppm
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:18.042359 5343 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:18.042893 5343 webserver.cc:460] Webserver started at http://127.2.251.3:42069/ using document root <none> and password file <none>
I20260811 02:37:18.043123 5343 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:18.043177 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:18.043291 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:18.044479 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "cea55b8ecf0f49a28fb1fa6261f93492"
format_stamp: "Formatted at 2026-08-11 02:37:18 on dist-test-slave-cvwc"
server_key: "5bc16e8d1d4a0d973696557edf8a1bed"
server_key_iv: "a59e182a3b6af33dc2d2ddb5022483eb"
server_key_version: "encryptionkey@0"
I20260811 02:37:18.044884 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "cea55b8ecf0f49a28fb1fa6261f93492"
format_stamp: "Formatted at 2026-08-11 02:37:18 on dist-test-slave-cvwc"
server_key: "5bc16e8d1d4a0d973696557edf8a1bed"
server_key_iv: "a59e182a3b6af33dc2d2ddb5022483eb"
server_key_version: "encryptionkey@0"
I20260811 02:37:18.046675 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:37:18.047498 5365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:18.047684 5343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:18.047770 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "cea55b8ecf0f49a28fb1fa6261f93492"
format_stamp: "Formatted at 2026-08-11 02:37:18 on dist-test-slave-cvwc"
server_key: "5bc16e8d1d4a0d973696557edf8a1bed"
server_key_iv: "a59e182a3b6af33dc2d2ddb5022483eb"
server_key_version: "encryptionkey@0"
I20260811 02:37:18.047838 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:18.059199 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:18.061923 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:18.062076 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:18.062660 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:18.062703 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:18.062726 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:18.062738 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:18.069676 5343 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:44465
I20260811 02:37:18.069802 5546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:44465 every 8 connection(s)
I20260811 02:37:18.070151 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415838, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:18.073920 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.063113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:59039 (local address 127.2.251.62:34429)
0811 02:37:18.063182 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.063184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.063483 (+ 299us) server_negotiation.cc:407] Connection header received
0811 02:37:18.064163 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.064165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.064180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.064262 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:18.065513 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.066285 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.067841 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.068138 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.068221 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.071107 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:18.071109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:18.071109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:18.071117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:18.072540 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.072949 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.072950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.072951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.073007 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.073233 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.073234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.073234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.073353 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:18.073410 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.073688 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.073814 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 02:37:18.074118 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.063113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.2.251.3:59039)
0811 02:37:18.063343 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.063389 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.063990 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.064523 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.064531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.065070 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:18.065379 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.065391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.066400 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.066401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.067741 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.067744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.068264 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.068266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.068441 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.069087 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:18.069099 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:18.071007 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:18.072638 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:18.072645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:18.072648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:18.072873 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:18.073065 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:18.073066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:18.073066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:18.073173 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:18.073415 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:18.073423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:18.073563 (+ 140us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.073655 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.073851 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260811 02:37:18.074620 5547 heartbeater.cc:347] Connected to a master server at 127.2.251.62:34429
I20260811 02:37:18.074718 5547 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:18.074931 5547 heartbeater.cc:511] Master 127.2.251.62:34429 requested a full tablet report, sending...
I20260811 02:37:18.075353 4779 ts_manager.cc:194] Registered new tserver with Master: cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465)
I20260811 02:37:18.075856 4779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:59039
I20260811 02:37:18.077010 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5343
I20260811 02:37:18.077178 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:18.077355 3052 external_mini_cluster.cc:1428] Setting key 71eb44a7376027bd1cbc7f54f5a031c7
I20260811 02:37:18.078256 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:37:18.082077 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.075752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:33863 (local address 127.2.251.62:34429)
0811 02:37:18.075794 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.075795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.075874 (+ 79us) server_negotiation.cc:407] Connection header received
0811 02:37:18.075941 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.075942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.075952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.076007 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:18.076386 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.076867 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.077909 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.078198 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.078265 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.079439 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:18.079440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:18.079440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:18.079447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:18.080987 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.081304 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.081305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.081305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.081362 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.081532 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.081533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.081533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.081643 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:18.081702 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.081931 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.082009 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:37:18.082125 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.075665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.2.251.3:33863)
0811 02:37:18.075807 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.075818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.075882 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.076014 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.076015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.076208 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:18.076290 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.076293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.076981 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.076983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.077827 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.077831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.078290 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.078291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.078381 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.078887 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:18.078893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:18.079333 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:18.081076 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:18.081077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:18.081078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:18.081242 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:18.081414 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:18.081415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:18.081415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:18.081481 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:18.081699 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:18.081700 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:18.081785 (+ 85us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.081906 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.082021 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:18.102841 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.095928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.2.251.62:34429)
0811 02:37:18.095984 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.095986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.095993 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:37:18.096020 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.096021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.096030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.096078 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:18.096589 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.097084 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.097855 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.098042 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.098094 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.100262 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:18.100264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:18.100264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:18.100271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:18.101849 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.102188 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.102190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.102190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.102247 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.102392 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.102392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.102392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.102504 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:18.102574 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.102678 (+ 104us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.102762 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:37:18.104473 4779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:18.106195 4779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:18.113519 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.109036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.2.251.1:45681)
0811 02:37:18.109456 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.109460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.109469 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:18.109501 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.109502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.109515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.109584 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.109925 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.111118 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.112368 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.113075 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.113146 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.113211 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.113386 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.113429 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":100,"threads_started":1}
I20260811 02:37:18.113523 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.108895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45681 (local address 127.0.0.1:51348)
0811 02:37:18.109057 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.109103 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.109179 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.109604 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.109607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.109609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.109698 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.109700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.111220 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.111221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.112295 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.112297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.113169 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.113170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.113264 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.113281 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.113371 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.113429 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260811 02:37:18.113994 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.108895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44465 (local address 127.0.0.1:51918)
0811 02:37:18.109279 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.109287 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.109338 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.109798 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.109799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.109801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.109843 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.109844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.110818 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.110819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.112146 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.112149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.113757 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.113758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.113824 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.113829 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.113877 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.113918 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260811 02:37:18.114265 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.109289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.2.251.3:44465)
0811 02:37:18.109448 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.109453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.109465 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:18.109509 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.109511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.109528 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.109618 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.110061 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.110732 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.112266 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.113070 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.113149 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.113211 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.114114 (+ 903us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.114166 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260811 02:37:18.114408 4994 tablet_service.cc:1467] Processing CreateTablet for tablet 66ecee6d5275414da540fa71c3dc2cec (DEFAULT_TABLE table=test-table [id=ac058ea3de374f62a1a29f281d71986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:18.114866 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.109136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:42777 (local address 127.0.0.1:34132)
0811 02:37:18.109468 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.109478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.109528 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.109907 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.109908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.109910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.109972 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.109973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.112366 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.112366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.113095 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.113098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.114542 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.114543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.114619 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.114624 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.114679 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.114721 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":83,"threads_started":1}
I20260811 02:37:18.114928 4994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ecee6d5275414da540fa71c3dc2cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.114887 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 66ecee6d5275414da540fa71c3dc2cec (DEFAULT_TABLE table=test-table [id=ac058ea3de374f62a1a29f281d71986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:18.115132 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.109261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.2.251.2:42777)
0811 02:37:18.109759 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.109762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.109772 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:18.109808 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.109811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.109825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.109895 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.110036 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.111118 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.113189 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.113961 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.114026 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.114188 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.114993 (+ 805us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.115040 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":105,"threads_started":1}
I20260811 02:37:18.115240 5441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ecee6d5275414da540fa71c3dc2cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.115782 5231 tablet_service.cc:1467] Processing CreateTablet for tablet 66ecee6d5275414da540fa71c3dc2cec (DEFAULT_TABLE table=test-table [id=ac058ea3de374f62a1a29f281d71986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:18.116155 5231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ecee6d5275414da540fa71c3dc2cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.118351 5578 tablet_bootstrap.cc:492] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Bootstrap starting.
I20260811 02:37:18.118587 5579 tablet_bootstrap.cc:492] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Bootstrap starting.
I20260811 02:37:18.119139 5578 tablet_bootstrap.cc:654] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.119635 5578 log.cc:824] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:18.119719 5579 tablet_bootstrap.cc:654] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.119897 5581 tablet_bootstrap.cc:492] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Bootstrap starting.
I20260811 02:37:18.120060 5579 log.cc:824] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:18.120440 5578 tablet_bootstrap.cc:492] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: No bootstrap required, opened a new log
I20260811 02:37:18.120515 5578 ts_tablet_manager.cc:1397] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:18.120597 5581 tablet_bootstrap.cc:654] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.120894 5581 log.cc:824] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:18.120756 5579 tablet_bootstrap.cc:492] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: No bootstrap required, opened a new log
I20260811 02:37:18.120980 5579 ts_tablet_manager.cc:1397] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:18.121551 5581 tablet_bootstrap.cc:492] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: No bootstrap required, opened a new log
I20260811 02:37:18.121615 5581 ts_tablet_manager.cc:1397] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:18.122769 5578 raft_consensus.cc:348] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.123088 5578 raft_consensus.cc:374] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.123162 5578 raft_consensus.cc:729] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb636642614e449799bd41049493772f, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.123118 5581 raft_consensus.cc:348] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.123167 5579 raft_consensus.cc:348] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.123289 5581 raft_consensus.cc:374] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.123304 5579 raft_consensus.cc:374] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.123337 5579 raft_consensus.cc:729] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea55b8ecf0f49a28fb1fa6261f93492, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.123337 5581 raft_consensus.cc:729] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b5d205ffd14374826fa57b7a8cf22b, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.123318 5578 consensus_queue.cc:260] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.123440 5579 consensus_queue.cc:260] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.123476 5581 consensus_queue.cc:260] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.123616 5578 ts_tablet_manager.cc:1428] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:18.123662 5579 ts_tablet_manager.cc:1428] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:18.123741 5100 heartbeater.cc:503] Master 127.2.251.62:34429 was elected leader, sending a full tablet report...
I20260811 02:37:18.123776 5547 heartbeater.cc:503] Master 127.2.251.62:34429 was elected leader, sending a full tablet report...
I20260811 02:37:18.123791 5581 ts_tablet_manager.cc:1428] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:18.124044 5332 heartbeater.cc:503] Master 127.2.251.62:34429 was elected leader, sending a full tablet report...
I20260811 02:37:18.146864 5592 raft_consensus.cc:482] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:18.147009 5592 raft_consensus.cc:504] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.147409 5592 leader_election.cc:288] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465), eb636642614e449799bd41049493772f (127.2.251.1:45681)
W20260811 02:37:18.148103 5333 tablet.cc:2369] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:18.150416 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.147754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:39803 (local address 127.2.251.1:45681)
0811 02:37:18.147808 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.147809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.147821 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:18.147853 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.147854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.147867 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.147939 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.148122 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.148713 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.149624 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.150079 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.150142 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.150154 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.150317 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.150357 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:37:18.150411 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.147648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45681 (local address 127.2.251.2:39803)
0811 02:37:18.147697 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.147708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.147756 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.147956 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.147957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.147959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.148036 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.148039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.148791 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.148793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.149532 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.149534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.150151 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.150152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.150226 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.150229 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.150280 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.150335 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260811 02:37:18.150722 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.147648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44465 (local address 127.2.251.2:57417)
0811 02:37:18.147970 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.147979 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.148065 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.148268 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.148268 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.148270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.148511 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.148514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.149128 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.149129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.149902 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.149904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.150450 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.150451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.150538 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.150543 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.150595 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.150630 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":153,"thread_start_us":116,"threads_started":1}
I20260811 02:37:18.150766 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.147738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:57417 (local address 127.2.251.3:44465)
0811 02:37:18.147799 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.147801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.148036 (+ 235us) server_negotiation.cc:407] Connection header received
0811 02:37:18.148155 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.148156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.148172 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.148202 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.148587 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.149052 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.149981 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.150350 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.150408 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.150419 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.150653 (+ 234us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.150703 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:37:18.150755 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ecee6d5275414da540fa71c3dc2cec" candidate_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb636642614e449799bd41049493772f" is_pre_election: true
I20260811 02:37:18.150925 5030 raft_consensus.cc:2454] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b5d205ffd14374826fa57b7a8cf22b in term 0.
I20260811 02:37:18.151157 5143 leader_election.cc:302] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b5d205ffd14374826fa57b7a8cf22b, eb636642614e449799bd41049493772f; no voters:
I20260811 02:37:18.151280 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ecee6d5275414da540fa71c3dc2cec" candidate_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" is_pre_election: true
I20260811 02:37:18.151335 5592 raft_consensus.cc:2781] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:18.151418 5592 raft_consensus.cc:482] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:18.151453 5592 raft_consensus.cc:3037] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.151463 5468 raft_consensus.cc:2454] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b5d205ffd14374826fa57b7a8cf22b in term 0.
I20260811 02:37:18.152742 5592 raft_consensus.cc:504] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.153019 5592 leader_election.cc:288] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [CANDIDATE]: Term 1 election: Requested vote from peers cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465), eb636642614e449799bd41049493772f (127.2.251.1:45681)
I20260811 02:37:18.153064 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ecee6d5275414da540fa71c3dc2cec" candidate_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea55b8ecf0f49a28fb1fa6261f93492"
I20260811 02:37:18.153148 5468 raft_consensus.cc:3037] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.153321 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ecee6d5275414da540fa71c3dc2cec" candidate_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb636642614e449799bd41049493772f"
I20260811 02:37:18.153403 5030 raft_consensus.cc:3037] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.153927 5468 raft_consensus.cc:2454] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b5d205ffd14374826fa57b7a8cf22b in term 1.
I20260811 02:37:18.154122 5142 leader_election.cc:302] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b5d205ffd14374826fa57b7a8cf22b, cea55b8ecf0f49a28fb1fa6261f93492; no voters:
I20260811 02:37:18.154146 5030 raft_consensus.cc:2454] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b5d205ffd14374826fa57b7a8cf22b in term 1.
I20260811 02:37:18.154309 5592 raft_consensus.cc:2781] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:18.154497 5592 raft_consensus.cc:686] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 1 LEADER]: Becoming Leader. State: Replica: a4b5d205ffd14374826fa57b7a8cf22b, State: Running, Role: LEADER
I20260811 02:37:18.154595 5592 consensus_queue.cc:237] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } }
I20260811 02:37:18.155661 4779 catalog_manager.cc:5495] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2). New cstate: current_term: 1 leader_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } health_report { overall_health: HEALTHY } } }
I20260811 02:37:18.164970 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.162287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.2.251.2:42777)
0811 02:37:18.162340 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.162342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.162351 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:18.162414 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.162414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.162434 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.162493 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:18.162724 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.163235 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.164055 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.164247 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.164314 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.164632 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.164739 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.164810 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.164857 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.164884 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:37:18.167042 5468 raft_consensus.cc:1261] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Refusing update from remote peer a4b5d205ffd14374826fa57b7a8cf22b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:18.167042 5030 raft_consensus.cc:1261] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Refusing update from remote peer a4b5d205ffd14374826fa57b7a8cf22b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:18.167574 5594 consensus_queue.cc:1035] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:18.169689 5594 consensus_queue.cc:1035] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:18.174664 4779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:18.174827 4779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:18.176056 4779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: Sending DeleteTablet for 3 replicas of tablet 66ecee6d5275414da540fa71c3dc2cec
I20260811 02:37:18.176613 5231 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ecee6d5275414da540fa71c3dc2cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:18 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:34132
I20260811 02:37:18.176620 4994 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ecee6d5275414da540fa71c3dc2cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:18 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:51348
I20260811 02:37:18.176782 4779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:18.176980 4779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:18.176990 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ecee6d5275414da540fa71c3dc2cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:18 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:51918
I20260811 02:37:18.177315 5623 tablet_replica.cc:330] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: stopping tablet replica
I20260811 02:37:18.177443 5623 raft_consensus.cc:2229] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:18.177562 5623 raft_consensus.cc:2258] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:18.177753 5621 tablet_replica.cc:330] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: stopping tablet replica
I20260811 02:37:18.177847 5621 raft_consensus.cc:2229] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:18.177986 5621 raft_consensus.cc:2258] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:18.178376 4994 tablet_service.cc:1467] Processing CreateTablet for tablet cc448b1b2f874b87a8393a7634d14e88 (DEFAULT_TABLE table=test-table [id=58f7ae567d46420fadf5efe810df802f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:18.178454 5231 tablet_service.cc:1467] Processing CreateTablet for tablet cc448b1b2f874b87a8393a7634d14e88 (DEFAULT_TABLE table=test-table [id=58f7ae567d46420fadf5efe810df802f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:18.178500 5441 tablet_service.cc:1467] Processing CreateTablet for tablet cc448b1b2f874b87a8393a7634d14e88 (DEFAULT_TABLE table=test-table [id=58f7ae567d46420fadf5efe810df802f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:18.178539 4994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc448b1b2f874b87a8393a7634d14e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.178587 5231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc448b1b2f874b87a8393a7634d14e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.178593 5441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc448b1b2f874b87a8393a7634d14e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.179925 5579 tablet_bootstrap.cc:492] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Bootstrap starting.
I20260811 02:37:18.179946 5578 tablet_bootstrap.cc:492] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Bootstrap starting.
I20260811 02:37:18.180294 5581 tablet_bootstrap.cc:492] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Bootstrap starting.
I20260811 02:37:18.180505 5579 tablet_bootstrap.cc:654] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.180613 5578 tablet_bootstrap.cc:654] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.180704 5581 tablet_bootstrap.cc:654] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.181196 5579 tablet_bootstrap.cc:492] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: No bootstrap required, opened a new log
I20260811 02:37:18.181242 5579 ts_tablet_manager.cc:1397] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:18.181308 5623 ts_tablet_manager.cc:1905] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:18.181375 5579 raft_consensus.cc:348] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.181443 5579 raft_consensus.cc:374] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.181457 5579 raft_consensus.cc:729] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea55b8ecf0f49a28fb1fa6261f93492, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.181501 5579 consensus_queue.cc:260] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.181589 5579 ts_tablet_manager.cc:1428] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:18.182199 5622 tablet_replica.cc:330] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: stopping tablet replica
I20260811 02:37:18.182199 5621 ts_tablet_manager.cc:1905] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:18.182317 5622 raft_consensus.cc:2229] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:18.182440 5622 raft_consensus.cc:2258] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:18.182889 5622 ts_tablet_manager.cc:1905] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:18.183873 5581 tablet_bootstrap.cc:492] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: No bootstrap required, opened a new log
I20260811 02:37:18.183933 5581 ts_tablet_manager.cc:1397] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260811 02:37:18.184083 5581 raft_consensus.cc:348] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.184160 5581 raft_consensus.cc:374] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.184180 5581 raft_consensus.cc:729] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b5d205ffd14374826fa57b7a8cf22b, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.184247 5581 consensus_queue.cc:260] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.184405 5581 ts_tablet_manager.cc:1428] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:18.184511 5621 ts_tablet_manager.cc:1918] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:18.184571 5621 log.cc:1196] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/66ecee6d5275414da540fa71c3dc2cec
I20260811 02:37:18.184820 5621 ts_tablet_manager.cc:1939] T 66ecee6d5275414da540fa71c3dc2cec P a4b5d205ffd14374826fa57b7a8cf22b: Deleting consensus metadata
I20260811 02:37:18.185331 5622 ts_tablet_manager.cc:1918] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:18.185274 4754 catalog_manager.cc:4841] TS a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777): tablet 66ecee6d5275414da540fa71c3dc2cec (table test-table [id=ac058ea3de374f62a1a29f281d71986e]) successfully deleted
I20260811 02:37:18.185420 5622 log.cc:1196] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/66ecee6d5275414da540fa71c3dc2cec
I20260811 02:37:18.185663 5622 ts_tablet_manager.cc:1939] T 66ecee6d5275414da540fa71c3dc2cec P eb636642614e449799bd41049493772f: Deleting consensus metadata
I20260811 02:37:18.185957 5623 ts_tablet_manager.cc:1918] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:18.186043 5623 log.cc:1196] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/66ecee6d5275414da540fa71c3dc2cec
I20260811 02:37:18.186142 4755 catalog_manager.cc:4841] TS eb636642614e449799bd41049493772f (127.2.251.1:45681): tablet 66ecee6d5275414da540fa71c3dc2cec (table test-table [id=ac058ea3de374f62a1a29f281d71986e]) successfully deleted
I20260811 02:37:18.186302 5623 ts_tablet_manager.cc:1939] T 66ecee6d5275414da540fa71c3dc2cec P cea55b8ecf0f49a28fb1fa6261f93492: Deleting consensus metadata
I20260811 02:37:18.186702 4754 catalog_manager.cc:4841] TS cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465): tablet 66ecee6d5275414da540fa71c3dc2cec (table test-table [id=ac058ea3de374f62a1a29f281d71986e]) successfully deleted
I20260811 02:37:18.190284 5578 tablet_bootstrap.cc:492] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: No bootstrap required, opened a new log
I20260811 02:37:18.190372 5578 ts_tablet_manager.cc:1397] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260811 02:37:18.190536 5578 raft_consensus.cc:348] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.190625 5578 raft_consensus.cc:374] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.190646 5578 raft_consensus.cc:729] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb636642614e449799bd41049493772f, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.190691 5578 consensus_queue.cc:260] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.190783 5578 ts_tablet_manager.cc:1428] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260811 02:37:18.234253 5101 tablet.cc:2369] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:37:18.320765 5549 tablet.cc:2369] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:18.344718 5590 raft_consensus.cc:482] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:18.344833 5590 raft_consensus.cc:504] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.345256 5590 leader_election.cc:288] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777), cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465)
I20260811 02:37:18.348299 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.345801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:48563 (local address 127.2.251.3:44465)
0811 02:37:18.345855 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.345856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.345905 (+ 49us) server_negotiation.cc:407] Connection header received
0811 02:37:18.345991 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.345992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.346002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.346039 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.346186 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.346652 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.347605 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.347971 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.348032 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.348044 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.348191 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.348229 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:37:18.348295 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.345600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:37465 (local address 127.2.251.2:42777)
0811 02:37:18.345660 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.345662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.345736 (+ 74us) server_negotiation.cc:407] Connection header received
0811 02:37:18.345904 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.345906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.345918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.345968 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.346104 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.346589 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.347600 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.347972 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.348033 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.348045 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.348184 (+ 139us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.348223 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:37:18.348343 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.345531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:42777 (local address 127.2.251.1:37465)
0811 02:37:18.345652 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.345663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.345784 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.345975 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.345976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.345979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.346046 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.346048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.346674 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.346675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.347508 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.347511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.348051 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.348052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.348120 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.348124 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.348170 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.348217 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":43,"thread_start_us":60,"threads_started":1}
I20260811 02:37:18.348351 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.345705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44465 (local address 127.2.251.1:48563)
0811 02:37:18.345834 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.345844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.345925 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.346053 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.346054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.346056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.346128 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.346129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.346731 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.346732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.347515 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.347517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.348048 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.348049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.348128 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.348132 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.348177 (+ 45us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.348220 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260811 02:37:18.348667 5258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc448b1b2f874b87a8393a7634d14e88" candidate_uuid: "eb636642614e449799bd41049493772f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" is_pre_election: true
I20260811 02:37:18.348723 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc448b1b2f874b87a8393a7634d14e88" candidate_uuid: "eb636642614e449799bd41049493772f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" is_pre_election: true
I20260811 02:37:18.348814 5468 raft_consensus.cc:2454] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb636642614e449799bd41049493772f in term 0.
I20260811 02:37:18.348810 5258 raft_consensus.cc:2454] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb636642614e449799bd41049493772f in term 0.
I20260811 02:37:18.349000 4920 leader_election.cc:302] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b5d205ffd14374826fa57b7a8cf22b, eb636642614e449799bd41049493772f; no voters:
I20260811 02:37:18.349254 5590 raft_consensus.cc:2781] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:18.349371 5590 raft_consensus.cc:482] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:18.349390 5590 raft_consensus.cc:3037] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.350358 5590 raft_consensus.cc:504] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.350497 5590 leader_election.cc:288] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [CANDIDATE]: Term 1 election: Requested vote from peers a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777), cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465)
I20260811 02:37:18.350673 5258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc448b1b2f874b87a8393a7634d14e88" candidate_uuid: "eb636642614e449799bd41049493772f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5d205ffd14374826fa57b7a8cf22b"
I20260811 02:37:18.350703 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc448b1b2f874b87a8393a7634d14e88" candidate_uuid: "eb636642614e449799bd41049493772f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea55b8ecf0f49a28fb1fa6261f93492"
I20260811 02:37:18.350765 5468 raft_consensus.cc:3037] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.350767 5258 raft_consensus.cc:3037] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.351409 5468 raft_consensus.cc:2454] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb636642614e449799bd41049493772f in term 1.
I20260811 02:37:18.351542 5258 raft_consensus.cc:2454] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb636642614e449799bd41049493772f in term 1.
I20260811 02:37:18.351577 4920 leader_election.cc:302] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea55b8ecf0f49a28fb1fa6261f93492, eb636642614e449799bd41049493772f; no voters:
I20260811 02:37:18.351672 5590 raft_consensus.cc:2781] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:18.351804 5590 raft_consensus.cc:686] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 1 LEADER]: Becoming Leader. State: Replica: eb636642614e449799bd41049493772f, State: Running, Role: LEADER
I20260811 02:37:18.351881 5590 consensus_queue.cc:237] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.352607 4779 catalog_manager.cc:5495] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f reported cstate change: term changed from 0 to 1, leader changed from <none> to eb636642614e449799bd41049493772f (127.2.251.1). New cstate: current_term: 1 leader_uuid: "eb636642614e449799bd41049493772f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } health_report { overall_health: UNKNOWN } } }
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:18.394912 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.386906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.2.251.62:34429)
0811 02:37:18.387006 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.387008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.387018 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:18.387064 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.387065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.387075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.387131 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:18.387672 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.388164 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.388930 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.389127 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.389195 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.391908 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:18.391910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:18.391910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:18.391918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:18.393594 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.394077 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.394079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.394079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.394142 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.394419 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.394420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.394421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.394562 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:18.394626 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.394751 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.394832 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 02:37:18.395026 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.386764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34429 (local address 127.0.0.1:59806)
0811 02:37:18.386935 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.386948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.387012 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.387169 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.387170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.387497 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:18.387550 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.387553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.388252 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.388253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.388817 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.388818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.389251 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.389252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.389303 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.389955 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:18.389960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:18.391770 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:18.393693 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:18.393703 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:18.393706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:18.393987 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:18.394255 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:18.394256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:18.394256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:18.394328 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:18.394621 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:18.394630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:18.394680 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.394733 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.394891 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":48,"thread_start_us":80,"threads_started":1}
I20260811 02:37:18.397544 4779 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 02:37:18.397786 4779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:18.400506 4994 tablet_service.cc:1467] Processing CreateTablet for tablet 25480bd8a92040859bbb0013a480fafd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9589e050e6604a29ba772c4d1e53815c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:18.400569 5231 tablet_service.cc:1467] Processing CreateTablet for tablet 25480bd8a92040859bbb0013a480fafd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9589e050e6604a29ba772c4d1e53815c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:18.400600 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 25480bd8a92040859bbb0013a480fafd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9589e050e6604a29ba772c4d1e53815c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:18.400735 5231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25480bd8a92040859bbb0013a480fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.400739 5441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25480bd8a92040859bbb0013a480fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.400738 4994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25480bd8a92040859bbb0013a480fafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:18.402486 5578 tablet_bootstrap.cc:492] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: Bootstrap starting.
I20260811 02:37:18.402982 5578 tablet_bootstrap.cc:654] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.403741 5578 tablet_bootstrap.cc:492] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: No bootstrap required, opened a new log
I20260811 02:37:18.403784 5578 ts_tablet_manager.cc:1397] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:18.403913 5578 raft_consensus.cc:348] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.403972 5578 raft_consensus.cc:374] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.404003 5578 raft_consensus.cc:729] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb636642614e449799bd41049493772f, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.404053 5578 consensus_queue.cc:260] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.404166 5578 ts_tablet_manager.cc:1428] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:18.404445 5579 tablet_bootstrap.cc:492] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: Bootstrap starting.
I20260811 02:37:18.404495 5590 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.404590 5590 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.404904 5579 tablet_bootstrap.cc:654] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.404899 5581 tablet_bootstrap.cc:492] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: Bootstrap starting.
I20260811 02:37:18.405233 5581 tablet_bootstrap.cc:654] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:18.405609 5579 tablet_bootstrap.cc:492] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: No bootstrap required, opened a new log
I20260811 02:37:18.405753 5579 ts_tablet_manager.cc:1397] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:18.405956 5579 raft_consensus.cc:348] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.406131 5579 raft_consensus.cc:374] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.406153 5579 raft_consensus.cc:729] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea55b8ecf0f49a28fb1fa6261f93492, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.406193 5581 tablet_bootstrap.cc:492] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: No bootstrap required, opened a new log
I20260811 02:37:18.406204 5579 consensus_queue.cc:260] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.406246 5581 ts_tablet_manager.cc:1397] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:18.406330 5579 ts_tablet_manager.cc:1428] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:18.406387 5581 raft_consensus.cc:348] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.406466 5581 raft_consensus.cc:374] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:18.406483 5581 raft_consensus.cc:729] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b5d205ffd14374826fa57b7a8cf22b, State: Initialized, Role: FOLLOWER
I20260811 02:37:18.406422 5613 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.406533 5613 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.406531 5581 consensus_queue.cc:260] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.406633 5581 ts_tablet_manager.cc:1428] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:18.406658 5594 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.406788 5594 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.582149 5613 raft_consensus.cc:482] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:18.582320 5613 raft_consensus.cc:504] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.582733 5613 leader_election.cc:288] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb636642614e449799bd41049493772f (127.2.251.1:45681), a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777)
I20260811 02:37:18.585989 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.582931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:42777 (local address 127.2.251.3:47527)
0811 02:37:18.583058 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.583069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.583127 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.583260 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.583261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.583263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.583330 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.583333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.583988 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.583989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.584905 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.584910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.585692 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.585693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.585796 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.585800 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.585846 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.585893 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260811 02:37:18.585994 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.583067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:47527 (local address 127.2.251.2:42777)
0811 02:37:18.583123 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.583125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.583135 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:18.583190 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.583191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.583205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.583252 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.583388 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.583884 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.585048 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.585605 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.585690 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.585703 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.585874 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.585920 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:37:18.586133 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.582936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45681 (local address 127.2.251.3:55355)
0811 02:37:18.583089 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.583100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.583164 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.583299 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.583299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.583302 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:18.583419 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.583423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.584366 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.584367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.585139 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.585142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.585902 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.585903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.585972 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.585976 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.586031 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.586067 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260811 02:37:18.586154 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.583015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:55355 (local address 127.2.251.1:45681)
0811 02:37:18.583063 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.583065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.583164 (+ 99us) server_negotiation.cc:407] Connection header received
0811 02:37:18.583220 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.583221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.583231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.583283 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:18.583562 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.584250 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.585262 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.585814 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.585886 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.585896 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.586044 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.586088 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:37:18.586441 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25480bd8a92040859bbb0013a480fafd" candidate_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb636642614e449799bd41049493772f" is_pre_election: true
I20260811 02:37:18.586555 5030 raft_consensus.cc:2454] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea55b8ecf0f49a28fb1fa6261f93492 in term 0.
I20260811 02:37:18.586615 5258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25480bd8a92040859bbb0013a480fafd" candidate_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" is_pre_election: true
I20260811 02:37:18.586714 5258 raft_consensus.cc:2454] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea55b8ecf0f49a28fb1fa6261f93492 in term 0.
I20260811 02:37:18.586748 5367 leader_election.cc:302] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea55b8ecf0f49a28fb1fa6261f93492, eb636642614e449799bd41049493772f; no voters:
I20260811 02:37:18.586898 5613 raft_consensus.cc:2781] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:18.586949 5613 raft_consensus.cc:482] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:18.586964 5613 raft_consensus.cc:3037] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.587648 5613 raft_consensus.cc:504] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.587771 5613 leader_election.cc:288] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [CANDIDATE]: Term 1 election: Requested vote from peers eb636642614e449799bd41049493772f (127.2.251.1:45681), a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777)
I20260811 02:37:18.587939 5258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25480bd8a92040859bbb0013a480fafd" candidate_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5d205ffd14374826fa57b7a8cf22b"
I20260811 02:37:18.587972 5030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25480bd8a92040859bbb0013a480fafd" candidate_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb636642614e449799bd41049493772f"
I20260811 02:37:18.588011 5258 raft_consensus.cc:3037] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.588037 5030 raft_consensus.cc:3037] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:18.588631 5258 raft_consensus.cc:2454] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea55b8ecf0f49a28fb1fa6261f93492 in term 1.
I20260811 02:37:18.588644 5030 raft_consensus.cc:2454] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea55b8ecf0f49a28fb1fa6261f93492 in term 1.
I20260811 02:37:18.588819 5367 leader_election.cc:302] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea55b8ecf0f49a28fb1fa6261f93492, eb636642614e449799bd41049493772f; no voters:
I20260811 02:37:18.588905 5613 raft_consensus.cc:2781] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:18.588976 5613 raft_consensus.cc:686] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 LEADER]: Becoming Leader. State: Replica: cea55b8ecf0f49a28fb1fa6261f93492, State: Running, Role: LEADER
I20260811 02:37:18.589044 5613 consensus_queue.cc:237] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } }
I20260811 02:37:18.589473 5591 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: TxnStatusTablet state changed. Reason: New leader cea55b8ecf0f49a28fb1fa6261f93492. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.589640 5591 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:18.589850 5657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 02:37:18.589963 5657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 02:37:18.590014 4776 catalog_manager.cc:5495] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 reported cstate change: term changed from 0 to 1, leader changed from <none> to cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3). New cstate: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } health_report { overall_health: HEALTHY } } }
I20260811 02:37:18.636421 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.633762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.2.251.3:44465)
0811 02:37:18.633827 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.633830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.633839 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:18.633911 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.633913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.633943 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.634000 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:18.634127 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.634612 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.635455 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.635660 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.635720 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.635920 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.636049 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.636226 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.636300 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.636326 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 02:37:18.636425 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.633663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44465 (local address 127.0.0.1:51922)
0811 02:37:18.633758 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.633768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.633821 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.634012 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.634013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.634014 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:18.634072 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.634074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.634687 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.634688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.635348 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.635352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.635730 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.635731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.635808 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.635826 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.636206 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.636228 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.636291 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.636351 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 02:37:18.637813 5030 raft_consensus.cc:1261] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Refusing update from remote peer cea55b8ecf0f49a28fb1fa6261f93492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:18.638011 5258 raft_consensus.cc:1261] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Refusing update from remote peer cea55b8ecf0f49a28fb1fa6261f93492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:18.638135 5591 consensus_queue.cc:1035] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:18.638373 5591 consensus_queue.cc:1035] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:18.638933 5590 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: TxnStatusTablet state changed. Reason: New leader cea55b8ecf0f49a28fb1fa6261f93492. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.639029 5590 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.639083 5594 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: TxnStatusTablet state changed. Reason: New leader cea55b8ecf0f49a28fb1fa6261f93492. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.639164 5594 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.640089 5613 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.640173 5613 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:18.640213 5613 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.640250 5613 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P cea55b8ecf0f49a28fb1fa6261f93492: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:18.640228 5600 mvcc.cc:205] Tried to move back new op lower bound from 7317159275058216960 to 7317159274861297664. Current Snapshot: MvccSnapshot[applied={T|T < 7317159275058216960}]
I20260811 02:37:18.640903 5601 mvcc.cc:205] Tried to move back new op lower bound from 7317159275058216960 to 7317159274861297664. Current Snapshot: MvccSnapshot[applied={T|T < 7317159275058216960 or (T in {7317159275058216960})}]
I20260811 02:37:18.641026 5594 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.641096 5594 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P a4b5d205ffd14374826fa57b7a8cf22b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.641255 5590 tablet_replica.cc:439] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb636642614e449799bd41049493772f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45681 } } peers { permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 } } peers { permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 } } }
I20260811 02:37:18.641323 5590 tablet_replica.cc:442] T 25480bd8a92040859bbb0013a480fafd P eb636642614e449799bd41049493772f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:18.641393 5607 mvcc.cc:205] Tried to move back new op lower bound from 7317159275058216960 to 7317159274861297664. Current Snapshot: MvccSnapshot[applied={T|T < 7317159275058216960 or (T in {7317159275058216960})}]
I20260811 02:37:18.645296 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.642384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.2.251.1:45681)
0811 02:37:18.642432 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.642434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.642493 (+ 59us) server_negotiation.cc:407] Connection header received
0811 02:37:18.642584 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.642585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.642608 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.642683 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:18.642802 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.643307 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.644418 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.644636 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.644763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.644926 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.645036 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.645134 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.645180 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.645202 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:18.651793 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.648651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:33483 (local address 127.2.251.3:44465)
0811 02:37:18.648701 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.648703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.648970 (+ 267us) server_negotiation.cc:407] Connection header received
0811 02:37:18.649026 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.649027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.649040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.649111 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:18.649237 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.650020 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.650935 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.651232 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.651303 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.651546 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.651591 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.651636 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.651701 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.651724 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:18.651866 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.648556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44465 (local address 127.2.251.1:33483)
0811 02:37:18.648874 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.648888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.648969 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.649115 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.649116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.649117 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:18.649181 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.649182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.650138 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.650141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.650820 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.650823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.651345 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.651346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.651422 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.651442 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.651649 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.651650 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.651697 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.651750 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260811 02:37:18.654552 5468 raft_consensus.cc:1261] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Refusing update from remote peer eb636642614e449799bd41049493772f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:18.654579 5258 raft_consensus.cc:1261] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Refusing update from remote peer eb636642614e449799bd41049493772f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:18.654898 5590 consensus_queue.cc:1035] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b5d205ffd14374826fa57b7a8cf22b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:18.655099 5638 consensus_queue.cc:1035] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea55b8ecf0f49a28fb1fa6261f93492" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:18.664697 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.661948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:57555 (local address 127.2.251.1:45681)
0811 02:37:18.661991 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.661993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.662142 (+ 149us) server_negotiation.cc:407] Connection header received
0811 02:37:18.662193 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.662194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.662204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.662247 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:18.662363 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.662842 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.663710 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.663981 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.664045 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.664222 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.664267 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.664329 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.664583 (+ 254us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.664610 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 02:37:18.664762 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.661913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45681 (local address 127.2.251.3:57555)
0811 02:37:18.662076 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:18.662087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:18.662144 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:18.662249 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:18.662250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:18.662251 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:18.662307 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.662309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.662918 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.662919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.663634 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:18.663637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.664082 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.664083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:18.664130 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.664146 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:18.664430 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:18.664439 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 02:37:18.664566 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 02:37:18.664654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260811 02:37:18.675529 4779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:18.675680 4779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:18.676595 4779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0d963c7183b41fca76c3881c5490302: Sending DeleteTablet for 3 replicas of tablet cc448b1b2f874b87a8393a7634d14e88
I20260811 02:37:18.676992 5231 tablet_service.cc:1514] Processing DeleteTablet for tablet cc448b1b2f874b87a8393a7634d14e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:18 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:34132
I20260811 02:37:18.677059 4994 tablet_service.cc:1514] Processing DeleteTablet for tablet cc448b1b2f874b87a8393a7634d14e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:18 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:51348
I20260811 02:37:18.677171 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet cc448b1b2f874b87a8393a7634d14e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:18 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:51918
I20260811 02:37:18.677289 5623 tablet_replica.cc:330] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: stopping tablet replica
I20260811 02:37:18.677297 5621 tablet_replica.cc:330] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: stopping tablet replica
I20260811 02:37:18.677354 5623 raft_consensus.cc:2229] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:18.677368 5621 raft_consensus.cc:2229] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:18.677402 5623 raft_consensus.cc:2258] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:18.677405 5621 raft_consensus.cc:2258] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:18.677479 5622 tablet_replica.cc:330] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: stopping tablet replica
I20260811 02:37:18.677536 5622 raft_consensus.cc:2229] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:18.677634 5622 raft_consensus.cc:2258] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:18.678198 5621 ts_tablet_manager.cc:1905] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:18.680022 5621 ts_tablet_manager.cc:1918] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:18.680091 5621 log.cc:1196] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/cc448b1b2f874b87a8393a7634d14e88
I20260811 02:37:18.680305 5621 ts_tablet_manager.cc:1939] T cc448b1b2f874b87a8393a7634d14e88 P a4b5d205ffd14374826fa57b7a8cf22b: Deleting consensus metadata
I20260811 02:37:18.680855 4754 catalog_manager.cc:4841] TS a4b5d205ffd14374826fa57b7a8cf22b (127.2.251.2:42777): tablet cc448b1b2f874b87a8393a7634d14e88 (table test-table [id=58f7ae567d46420fadf5efe810df802f]) successfully deleted
I20260811 02:37:18.681180 5622 ts_tablet_manager.cc:1905] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:18.681876 5623 ts_tablet_manager.cc:1905] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:18.683017 5622 ts_tablet_manager.cc:1918] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:18.683081 5622 log.cc:1196] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/cc448b1b2f874b87a8393a7634d14e88
I20260811 02:37:18.683316 5622 ts_tablet_manager.cc:1939] T cc448b1b2f874b87a8393a7634d14e88 P eb636642614e449799bd41049493772f: Deleting consensus metadata
I20260811 02:37:18.683607 5623 ts_tablet_manager.cc:1918] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:18.683666 5623 log.cc:1196] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/cc448b1b2f874b87a8393a7634d14e88
I20260811 02:37:18.683751 4755 catalog_manager.cc:4841] TS eb636642614e449799bd41049493772f (127.2.251.1:45681): tablet cc448b1b2f874b87a8393a7634d14e88 (table test-table [id=58f7ae567d46420fadf5efe810df802f]) successfully deleted
I20260811 02:37:18.683885 5623 ts_tablet_manager.cc:1939] T cc448b1b2f874b87a8393a7634d14e88 P cea55b8ecf0f49a28fb1fa6261f93492: Deleting consensus metadata
I20260811 02:37:18.684233 4754 catalog_manager.cc:4841] TS cea55b8ecf0f49a28fb1fa6261f93492 (127.2.251.3:44465): tablet cc448b1b2f874b87a8393a7634d14e88 (table test-table [id=58f7ae567d46420fadf5efe810df802f]) successfully deleted
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415838, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:18.697503 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.684541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.2.251.1:45681)
0811 02:37:18.684588 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.684590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.685925 (+ 1335us) server_negotiation.cc:407] Connection header received
0811 02:37:18.685989 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.685990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.686004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.686083 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:18.686879 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.687604 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.688731 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.689027 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.689104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.693685 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:18.693697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:18.693698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:18.693747 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:18.696038 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.696513 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.696518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.696518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.696611 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.696829 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.696830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.696830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.697024 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:18.697175 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.697276 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.697375 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260811 02:37:18.698014 5093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51356
I20260811 02:37:18.707862 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:18.699381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.2.251.62:34429)
0811 02:37:18.699436 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:18.699438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:18.699719 (+ 281us) server_negotiation.cc:407] Connection header received
0811 02:37:18.699772 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:18.699773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:18.699785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:18.699854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:18.700485 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.701274 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.702391 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:18.702706 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:18.702813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:18.704626 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:18.704628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:18.704628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:18.704637 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:18.706555 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.706980 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.706982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.706982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.707056 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:18.707264 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:18.707265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:18.707266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:18.707421 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:18.707530 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:18.707650 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 02:37:18.707754 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260811 02:37:18.708352 4779 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59816
I20260811 02:37:18.709374 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4888
I20260811 02:37:18.722137 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5114
I20260811 02:37:18.744505 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5343
I20260811 02:37:18.761296 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4730
2026-08-11T02:37:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2753 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[5694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[5694](info): set up 2 sockets
Aug 11 02:37:18 dist-test-slave-cvwc krb5kdc[5694](info): commencing operation
krb5kdc: starting...
W20260811 02:37:20.000511 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:20.137068 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:44215
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:44215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:20.223578 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:20.223826 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:20.223863 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:20.225063 5716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:20.225116 5716 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 02:37:20.225139 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:20.225158 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:20.225180 5716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:20.225203 5716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:20.226832 5716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:44215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:44215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:20.227114 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:20.227355 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:20.232621 5727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.232613 5729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.232641 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:20.232883 5716 server_base.cc:1034] running on GCE node
I20260811 02:37:20.233328 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:20.233614 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:20.234794 5716 hybrid_clock.cc:648] HybridClock initialized: now 1786415840234750 us; error 58 us; skew 500 ppm
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:20.238202 5716 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:20.238816 5716 webserver.cc:460] Webserver started at http://127.2.251.62:36987/ using document root <none> and password file <none>
I20260811 02:37:20.239037 5716 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:20.239087 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:20.239215 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:20.240459 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "9a212ba0e14a2a2e55eba02a96a4972d"
server_key_iv: "30c43c1c5f648c8d7872f9d649c0e5fc"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.241089 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "9a212ba0e14a2a2e55eba02a96a4972d"
server_key_iv: "30c43c1c5f648c8d7872f9d649c0e5fc"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.243399 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:20.244292 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.244480 5716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:20.244571 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "9a212ba0e14a2a2e55eba02a96a4972d"
server_key_iv: "30c43c1c5f648c8d7872f9d649c0e5fc"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.244647 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:20.265357 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:20.268335 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:20.268508 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:20.273286 5716 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:44215
I20260811 02:37:20.273342 5849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:44215 every 8 connection(s)
I20260811 02:37:20.273666 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:20.274435 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.276803 5851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Bootstrap starting.
I20260811 02:37:20.277410 5851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.277669 5851 log.cc:824] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:20.278497 5851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: No bootstrap required, opened a new log
I20260811 02:37:20.280637 5851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44215 } }
I20260811 02:37:20.280802 5851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.280825 5851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c14e15b1bbe40de8ccb5bb32ac890ff, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.280995 5851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44215 } }
I20260811 02:37:20.281081 5851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:20.281104 5851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:20.281133 5851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:20.281674 5851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44215 } }
I20260811 02:37:20.281810 5851 leader_election.cc:302] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c14e15b1bbe40de8ccb5bb32ac890ff; no voters:
I20260811 02:37:20.282014 5851 leader_election.cc:288] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:20.282146 5855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:20.282332 5851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:20.282424 5855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [term 1 LEADER]: Becoming Leader. State: Replica: 2c14e15b1bbe40de8ccb5bb32ac890ff, State: Running, Role: LEADER
I20260811 02:37:20.282567 5855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44215 } }
I20260811 02:37:20.283123 5856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44215 } } }
I20260811 02:37:20.283154 5857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c14e15b1bbe40de8ccb5bb32ac890ff. Latest consensus state: current_term: 1 leader_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c14e15b1bbe40de8ccb5bb32ac890ff" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44215 } } }
I20260811 02:37:20.283277 5856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:20.283284 5857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:20.283571 5864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:20.283773 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5716
I20260811 02:37:20.283866 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:20.284070 3052 external_mini_cluster.cc:1428] Setting key b00b018acb6000047fc18a00bc8ebd07
I20260811 02:37:20.284159 5864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:20.285871 5864 catalog_manager.cc:1347] Generated new cluster ID: 3656a46f1b534527b2696173df7b3815
I20260811 02:37:20.285919 5864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:20.295795 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.284655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.2.251.62:44215)
0811 02:37:20.284934 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.284945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.284982 (+ 37us) server_negotiation.cc:407] Connection header received
0811 02:37:20.285107 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.285113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.285159 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.285294 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.286103 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.287498 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.288602 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.289227 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.289324 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.292199 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.292204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.292204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.292219 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.294172 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.294547 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.294551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.294551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.294660 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.294832 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.294833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.294834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.295152 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.295323 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.295411 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.295550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":84,"threads_started":1}
I20260811 02:37:20.303771 5864 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:20.304665 5864 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:20.312606 5864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Generated new TSK 0
I20260811 02:37:20.312835 5864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:20.346912 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44215
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:20.440614 5889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:20.440873 5889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:20.440905 5889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:20.442293 5889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:20.442353 5889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:20.442376 5889 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 02:37:20.442389 5889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:20.442405 5889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:20.444288 5889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:44215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:20.444594 5889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:20.444867 5889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:20.451512 5900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.451550 5902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.451524 5899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:20.451697 5889 server_base.cc:1034] running on GCE node
I20260811 02:37:20.451951 5889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:20.452224 5889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:20.453405 5889 hybrid_clock.cc:648] HybridClock initialized: now 1786415840453394 us; error 78 us; skew 500 ppm
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:20.456220 5889 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:20.456738 5889 webserver.cc:460] Webserver started at http://127.2.251.1:46303/ using document root <none> and password file <none>
I20260811 02:37:20.456969 5889 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:20.457022 5889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:20.457150 5889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:20.458320 5889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "136b0fee7e7f4e81b9dcd8b6c494a401"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "daf44bff8c9cf970685694230b0a4c72"
server_key_iv: "97d66255b7fc77d661f40a1136b9e58d"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.458700 5889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "136b0fee7e7f4e81b9dcd8b6c494a401"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "daf44bff8c9cf970685694230b0a4c72"
server_key_iv: "97d66255b7fc77d661f40a1136b9e58d"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.460841 5889 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:37:20.461829 5917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.462123 5889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:20.462291 5889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "136b0fee7e7f4e81b9dcd8b6c494a401"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "daf44bff8c9cf970685694230b0a4c72"
server_key_iv: "97d66255b7fc77d661f40a1136b9e58d"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.462394 5889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:20.477380 5889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:20.480299 5889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:20.480472 5889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:20.481083 5889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:20.481125 5889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.481156 5889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:20.481173 5889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:20.489450 5889 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:45821
I20260811 02:37:20.489602 6101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:45821 every 8 connection(s)
I20260811 02:37:20.489910 5889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:20.492573 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.481597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:42683 (local address 127.2.251.62:44215)
0811 02:37:20.481654 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.481656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.482098 (+ 442us) server_negotiation.cc:407] Connection header received
0811 02:37:20.482815 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.482816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.482830 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.482885 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.483824 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.484626 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.486211 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.486543 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.486633 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.489578 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.489580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.489580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.489589 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.491056 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.491512 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.491514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.491514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.491575 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.491912 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.491914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.491914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.492044 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.492110 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.492412 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.492504 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:37:20.492673 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.481562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.2.251.1:42683)
0811 02:37:20.481967 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.482008 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.482692 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.482981 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.482986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.483268 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.483551 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.483557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.484723 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.484724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.486096 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.486100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.486669 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.486670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.486832 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.487500 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.487509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.489478 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.491151 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.491158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.491160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.491407 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.491661 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.491662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.491662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.491811 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.492112 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.492118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.492247 (+ 129us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.492341 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.492480 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":324,"threads_started":1}
I20260811 02:37:20.493198 6102 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44215
I20260811 02:37:20.493294 6102 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:20.493479 6102 heartbeater.cc:511] Master 127.2.251.62:44215 requested a full tablet report, sending...
I20260811 02:37:20.493927 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5889
I20260811 02:37:20.494025 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:20.494082 5771 ts_manager.cc:194] Registered new tserver with Master: 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821)
I20260811 02:37:20.494216 3052 external_mini_cluster.cc:1428] Setting key f0de61d5a6b6d35a427cbe0921206658
I20260811 02:37:20.494939 5771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:42683
I20260811 02:37:20.500788 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.494194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.2.251.1:57023)
0811 02:37:20.494383 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.494393 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.494633 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.494775 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.494775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.494966 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.495026 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.495028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.495715 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.495717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.496246 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.496248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.496700 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.496701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.496779 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.497279 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.497284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.497686 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.499628 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.499630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.499630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.499869 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.500096 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.500097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.500097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.500167 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.500424 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.500425 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.500478 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.500550 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.500675 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260811 02:37:20.500787 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.494258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:57023 (local address 127.2.251.62:44215)
0811 02:37:20.494298 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.494300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.494534 (+ 234us) server_negotiation.cc:407] Connection header received
0811 02:37:20.494695 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.494696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.494710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.494760 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.495122 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.495626 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.496320 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.496574 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.496639 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.497820 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.497822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.497822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.497830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.499513 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.499967 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.499968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.499969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.500036 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.500232 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.500233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.500233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.500360 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.500428 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.500581 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.500675 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:20.526741 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44215
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:20.619671 6116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:20.619916 6116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:20.619941 6116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:20.621088 6116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:20.621126 6116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:20.621146 6116 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 02:37:20.621156 6116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:20.621172 6116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:20.622799 6116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:44215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:20.623023 6116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:20.623220 6116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:20.628629 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.628629 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.628640 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:20.628890 6116 server_base.cc:1034] running on GCE node
I20260811 02:37:20.629202 6116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:20.629449 6116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:20.630630 6116 hybrid_clock.cc:648] HybridClock initialized: now 1786415840630588 us; error 55 us; skew 500 ppm
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:20.633529 6116 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:20.634058 6116 webserver.cc:460] Webserver started at http://127.2.251.2:46701/ using document root <none> and password file <none>
I20260811 02:37:20.634294 6116 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:20.634344 6116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:20.634486 6116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:20.635633 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "a354aa9503234e0bb8c9e456f11ba02d"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "05a4830f0265572bf4090753a24f3e65"
server_key_iv: "a3383a51a9ccb4a40a22c389df903b5f"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.636178 6116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "a354aa9503234e0bb8c9e456f11ba02d"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "05a4830f0265572bf4090753a24f3e65"
server_key_iv: "a3383a51a9ccb4a40a22c389df903b5f"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.638072 6116 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 02:37:20.639156 6138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.639410 6116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:20.639515 6116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "a354aa9503234e0bb8c9e456f11ba02d"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "05a4830f0265572bf4090753a24f3e65"
server_key_iv: "a3383a51a9ccb4a40a22c389df903b5f"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.639598 6116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:20.658445 6116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:20.661082 6116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:20.661242 6116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:20.661877 6116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:20.661931 6116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.661978 6116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:20.662009 6116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:20.669458 6116 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38707
I20260811 02:37:20.669534 6326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38707 every 8 connection(s)
I20260811 02:37:20.669891 6116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:20.672887 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.662634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:55403 (local address 127.2.251.62:44215)
0811 02:37:20.662729 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.662733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.663056 (+ 323us) server_negotiation.cc:407] Connection header received
0811 02:37:20.663836 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.663839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.663856 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.663942 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.664637 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.665392 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.667099 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.667391 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.667481 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.669978 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.669980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.669981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.669991 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.671450 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.671885 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.671886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.671887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.671948 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.672205 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.672206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.672206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.672332 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.672400 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.672691 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.672791 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 02:37:20.673058 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.662584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.2.251.2:55403)
0811 02:37:20.662841 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.662898 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.663665 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.664042 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.664047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.664322 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.664527 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.664532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.665508 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.665510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.666998 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.667003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.667494 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.667496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.667636 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.668225 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.668234 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.669877 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.671545 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.671553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.671556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.671801 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.672014 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.672014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.672015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.672135 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.672401 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.672409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.672555 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.672656 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.672834 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":122,"threads_started":1}
I20260811 02:37:20.673604 6327 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44215
I20260811 02:37:20.673697 6327 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:20.673768 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6116
I20260811 02:37:20.673879 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:20.673961 6327 heartbeater.cc:511] Master 127.2.251.62:44215 requested a full tablet report, sending...
I20260811 02:37:20.674098 3052 external_mini_cluster.cc:1428] Setting key 2f8ea925284f7d01de232d798865144f
I20260811 02:37:20.674371 5775 ts_manager.cc:194] Registered new tserver with Master: a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707)
I20260811 02:37:20.674865 5775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:55403
I20260811 02:37:20.682367 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.674839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52209 (local address 127.2.251.62:44215)
0811 02:37:20.674883 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.674885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.675014 (+ 129us) server_negotiation.cc:407] Connection header received
0811 02:37:20.675088 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.675089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.675099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.675145 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.675532 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.676174 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.677244 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.677532 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.677639 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.678892 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.678895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.678895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.678907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.680784 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.681195 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.681196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.681197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.681266 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.681508 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.681508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.681509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.681774 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.681848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.682146 (+ 298us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.682246 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 02:37:20.682391 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.674790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.2.251.2:52209)
0811 02:37:20.674949 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.674960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.675040 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.675148 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.675149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.675378 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.675439 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.675442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.676322 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.676323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.677064 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.677067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.677663 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.677664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.677843 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.678331 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.678337 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.678673 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.680903 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.680904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.680905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.681107 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.681359 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.681360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.681360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.681433 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.681895 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.681897 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.681988 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.682091 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.682254 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:20.710121 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44215
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:20.802696 6344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:20.802956 6344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:20.802999 6344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:20.804280 6344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:20.804337 6344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:20.804371 6344 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 02:37:20.804390 6344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:20.804426 6344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:20.806145 6344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:44215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:20.806417 6344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:20.806655 6344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:20.812062 6355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.812072 6357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:20.812083 6354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:20.812366 6344 server_base.cc:1034] running on GCE node
I20260811 02:37:20.812593 6344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:20.812886 6344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:20.814073 6344 hybrid_clock.cc:648] HybridClock initialized: now 1786415840814035 us; error 66 us; skew 500 ppm
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:20.816701 6344 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:20.817147 6344 webserver.cc:460] Webserver started at http://127.2.251.3:46841/ using document root <none> and password file <none>
I20260811 02:37:20.817461 6344 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:20.817538 6344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:20.817668 6344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:20.818850 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "19d64d99accb4ab7b10587c7b093cff0"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "120db76113356a0217c21f3ab31229e4"
server_key_iv: "446b3908fdbde85b2e5bd6ace3588ff8"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.819242 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "19d64d99accb4ab7b10587c7b093cff0"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "120db76113356a0217c21f3ab31229e4"
server_key_iv: "446b3908fdbde85b2e5bd6ace3588ff8"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.820778 6344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:20.821673 6368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.822063 6344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:20.822178 6344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "19d64d99accb4ab7b10587c7b093cff0"
format_stamp: "Formatted at 2026-08-11 02:37:20 on dist-test-slave-cvwc"
server_key: "120db76113356a0217c21f3ab31229e4"
server_key_iv: "446b3908fdbde85b2e5bd6ace3588ff8"
server_key_version: "encryptionkey@0"
I20260811 02:37:20.822263 6344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:20.837834 6344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:20.840354 6344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:20.840529 6344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:20.841151 6344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:20.841207 6344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.841245 6344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:20.841266 6344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:20.849444 6344 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:34959
I20260811 02:37:20.849545 6580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:34959 every 8 connection(s)
I20260811 02:37:20.849941 6344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:20.852674 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.841742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:51695 (local address 127.2.251.62:44215)
0811 02:37:20.841791 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.841793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.842026 (+ 233us) server_negotiation.cc:407] Connection header received
0811 02:37:20.842562 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.842563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.842576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.842628 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.843487 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.844242 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.845507 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.845862 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.845931 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.849134 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.849136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.849136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.849145 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.850918 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.851418 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.851420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.851420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.851509 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.851861 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.851862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.851862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.852025 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.852094 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.852476 (+ 382us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.852574 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:37:20.852851 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.841678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.2.251.3:51695)
0811 02:37:20.841897 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.841935 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.842457 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.842726 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.842730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.843074 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.843366 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.843374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.844358 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.844360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.845417 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.845422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.846046 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.846047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.846198 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.846880 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.846890 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.848995 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.851019 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.851026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.851029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.851318 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.851597 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.851598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.851598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.851772 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.852089 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.852097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.852302 (+ 205us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.852426 (+ 124us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.852618 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":127,"threads_started":1}
I20260811 02:37:20.853399 6582 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44215
I20260811 02:37:20.853480 6582 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:20.853674 6582 heartbeater.cc:511] Master 127.2.251.62:44215 requested a full tablet report, sending...
I20260811 02:37:20.854820 5771 ts_manager.cc:194] Registered new tserver with Master: 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:20.855502 5771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:51695
I20260811 02:37:20.856889 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6344
I20260811 02:37:20.856967 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:20.857159 3052 external_mini_cluster.cc:1428] Setting key 38279d4b391f40283de83510993803ce
I20260811 02:37:20.858702 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:37:20.862862 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.854682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:32809 (local address 127.2.251.62:44215)
0811 02:37:20.854725 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.854726 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.854917 (+ 191us) server_negotiation.cc:407] Connection header received
0811 02:37:20.855034 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.855035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.855048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.855107 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.855670 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.856366 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.857083 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.857358 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.857427 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.859131 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.859133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.859133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.859141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.861330 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.861993 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.861995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.861995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.862079 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.862294 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.862295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.862295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.862426 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.862494 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.862661 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.862760 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:37:20.862871 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.854632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.2.251.3:32809)
0811 02:37:20.854848 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.854860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.854946 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.855176 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.855177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.855396 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.855477 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.855479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.856477 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.856478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.856997 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.856999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.857572 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.857576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.857766 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.858542 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.858548 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.859024 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.861465 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.861467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.861468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.861885 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.862155 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.862156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.862156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.862231 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.862492 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.862492 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.862553 (+ 61us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.862627 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.862763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":144,"threads_started":1}
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:20.889451 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.880251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.2.251.62:44215)
0811 02:37:20.880317 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.880320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.880330 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:20.880360 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.880361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.880372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.880432 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.881099 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.881630 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.882481 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.882679 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.882760 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.886212 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.886214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.886214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.886224 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.887835 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.888323 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.888326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.888326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.888411 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.888782 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.888784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.888784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.888948 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.889047 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.889282 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.889376 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:37:20.891445 5771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:20.893241 5771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:20.901072 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.896826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.0.0.1:59572)
0811 02:37:20.897075 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.897084 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.897131 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.897695 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.897696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.897698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:20.897758 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.897759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.899084 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.899085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.899976 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.899979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.900828 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.900829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.900898 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.900910 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.900966 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.901006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":52,"thread_start_us":100,"threads_started":1}
I20260811 02:37:20.901160 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.897082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.2.251.1:45821)
0811 02:37:20.897546 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.897550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.897559 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:20.897595 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.897597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.897611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.897681 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:20.897827 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.898984 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.900078 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.900736 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.900828 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.900892 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.900999 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.901041 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":107,"threads_started":1}
I20260811 02:37:20.901901 5991 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd14302a4b145baa4f653aac8f9bf74 (DEFAULT_TABLE table=test-table [id=5e734fb81428436c814a2f1ec170ef90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:20.902120 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.896589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38707 (local address 127.0.0.1:37490)
0811 02:37:20.896790 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.896849 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.896916 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.897455 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.897456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.897458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:20.897517 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.897519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.898991 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.898993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.900329 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.900332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.901849 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.901851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.901921 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.901925 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.901999 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.902045 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":7,"thread_start_us":90,"threads_started":1}
I20260811 02:37:20.902436 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd14302a4b145baa4f653aac8f9bf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.902498 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.896589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34959 (local address 127.0.0.1:54270)
0811 02:37:20.896989 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.897000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.897062 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.897374 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.897377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.897379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:20.897464 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.897466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.899110 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.899110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.900330 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.900332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.902263 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.902265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.902327 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.902331 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.902388 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.902430 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":58,"thread_start_us":72,"threads_started":1}
I20260811 02:37:20.902570 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.896838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.2.251.3:34959)
0811 02:37:20.897177 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.897182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.897193 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:20.897244 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.897246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.897262 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.897342 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:20.897549 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.899009 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.901372 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.902084 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.902152 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.902223 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.902420 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.902467 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":99,"threads_started":1}
I20260811 02:37:20.903008 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.897074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.2.251.2:38707)
0811 02:37:20.897264 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.897269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.897278 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:20.897323 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.897325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.897349 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.897426 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:20.897595 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.898864 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.900428 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.901154 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.901250 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.901330 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.902031 (+ 701us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.902080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260811 02:37:20.903138 6445 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd14302a4b145baa4f653aac8f9bf74 (DEFAULT_TABLE table=test-table [id=5e734fb81428436c814a2f1ec170ef90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:20.903918 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd14302a4b145baa4f653aac8f9bf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.906533 6222 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd14302a4b145baa4f653aac8f9bf74 (DEFAULT_TABLE table=test-table [id=5e734fb81428436c814a2f1ec170ef90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:20.906854 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd14302a4b145baa4f653aac8f9bf74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.906931 6620 tablet_bootstrap.cc:492] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Bootstrap starting.
I20260811 02:37:20.907963 6620 tablet_bootstrap.cc:654] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.908308 6620 log.cc:824] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:20.909163 6620 tablet_bootstrap.cc:492] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: No bootstrap required, opened a new log
I20260811 02:37:20.909241 6620 ts_tablet_manager.cc:1397] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:20.910590 6623 tablet_bootstrap.cc:492] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Bootstrap starting.
I20260811 02:37:20.910884 6625 tablet_bootstrap.cc:492] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Bootstrap starting.
I20260811 02:37:20.911204 6623 tablet_bootstrap.cc:654] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.911293 6620 raft_consensus.cc:348] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.911471 6620 raft_consensus.cc:374] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.911525 6625 tablet_bootstrap.cc:654] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.911520 6620 raft_consensus.cc:729] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d64d99accb4ab7b10587c7b093cff0, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.911512 6623 log.cc:824] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:20.911671 6620 consensus_queue.cc:260] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.911849 6625 log.cc:824] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:20.911983 6620 ts_tablet_manager.cc:1428] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:20.912125 6582 heartbeater.cc:503] Master 127.2.251.62:44215 was elected leader, sending a full tablet report...
I20260811 02:37:20.912361 6623 tablet_bootstrap.cc:492] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: No bootstrap required, opened a new log
I20260811 02:37:20.912441 6623 ts_tablet_manager.cc:1397] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:20.912698 6625 tablet_bootstrap.cc:492] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: No bootstrap required, opened a new log
I20260811 02:37:20.912810 6625 ts_tablet_manager.cc:1397] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:20.914113 6623 raft_consensus.cc:348] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.914242 6623 raft_consensus.cc:374] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.914261 6623 raft_consensus.cc:729] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.914361 6623 consensus_queue.cc:260] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.914345 6625 raft_consensus.cc:348] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.914499 6625 raft_consensus.cc:374] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.914538 6625 raft_consensus.cc:729] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.914636 6623 ts_tablet_manager.cc:1428] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:37:20.914664 6327 heartbeater.cc:503] Master 127.2.251.62:44215 was elected leader, sending a full tablet report...
I20260811 02:37:20.914685 6625 consensus_queue.cc:260] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.914943 6625 ts_tablet_manager.cc:1428] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:20.915052 6102 heartbeater.cc:503] Master 127.2.251.62:44215 was elected leader, sending a full tablet report...
W20260811 02:37:20.920753 6328 tablet.cc:2369] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:20.931788 6631 raft_consensus.cc:482] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:20.931941 6631 raft_consensus.cc:504] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.932330 6631 leader_election.cc:288] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:20.935655 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.932553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.2.251.2:52647)
0811 02:37:20.932612 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.932626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.932686 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.932835 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.932836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.932839 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:20.932915 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.932917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.933898 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.933899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.934719 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.934722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.935422 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.935423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.935504 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.935508 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.935560 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.935596 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 02:37:20.935743 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.932651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52647 (local address 127.2.251.1:45821)
0811 02:37:20.932698 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.932700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.932710 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:20.932741 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.932742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.932754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.932809 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:20.933098 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.933794 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.934851 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.935319 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.935380 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.935389 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.935602 (+ 213us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.935657 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 02:37:20.935829 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.932683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:53313 (local address 127.2.251.3:34959)
0811 02:37:20.932731 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.932733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.932877 (+ 144us) server_negotiation.cc:407] Connection header received
0811 02:37:20.932946 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.932947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.932960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.933011 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:20.933406 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.934021 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.935044 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.935430 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.935506 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.935515 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.935710 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.935756 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:37:20.936012 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.932553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34959 (local address 127.2.251.2:53313)
0811 02:37:20.932806 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.932816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.932866 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.933100 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.933102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.933107 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:20.933246 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.933248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.934138 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.934140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.934944 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.934947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.935535 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.935536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.935622 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.935627 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.935702 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.935911 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":59,"thread_start_us":133,"threads_started":1}
I20260811 02:37:20.936048 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd14302a4b145baa4f653aac8f9bf74" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" is_pre_election: true
I20260811 02:37:20.936254 6019 raft_consensus.cc:2454] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 0.
I20260811 02:37:20.936376 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd14302a4b145baa4f653aac8f9bf74" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0" is_pre_election: true
I20260811 02:37:20.936524 6471 raft_consensus.cc:2454] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 0.
I20260811 02:37:20.936571 6140 leader_election.cc:302] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, a354aa9503234e0bb8c9e456f11ba02d; no voters:
I20260811 02:37:20.936728 6631 raft_consensus.cc:2781] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:20.936780 6631 raft_consensus.cc:482] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:20.936796 6631 raft_consensus.cc:3037] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:20.937901 6631 raft_consensus.cc:504] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.938123 6631 leader_election.cc:288] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 election: Requested vote from peers 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:20.938268 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd14302a4b145baa4f653aac8f9bf74" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401"
I20260811 02:37:20.938383 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd14302a4b145baa4f653aac8f9bf74" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0"
I20260811 02:37:20.938513 6471 raft_consensus.cc:3037] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:20.938445 6019 raft_consensus.cc:3037] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:20.939785 6471 raft_consensus.cc:2454] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 1.
I20260811 02:37:20.939783 6019 raft_consensus.cc:2454] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 1.
I20260811 02:37:20.939977 6140 leader_election.cc:302] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, a354aa9503234e0bb8c9e456f11ba02d; no voters:
I20260811 02:37:20.940083 6631 raft_consensus.cc:2781] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:20.940264 6631 raft_consensus.cc:686] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 1 LEADER]: Becoming Leader. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Running, Role: LEADER
I20260811 02:37:20.940459 6631 consensus_queue.cc:237] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.941406 5771 catalog_manager.cc:5495] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d reported cstate change: term changed from 0 to 1, leader changed from <none> to a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2). New cstate: current_term: 1 leader_uuid: "a354aa9503234e0bb8c9e456f11ba02d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
Aug 11 02:37:20 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415840, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:20.975551 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.966752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.2.251.62:44215)
0811 02:37:20.966815 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:20.966817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:20.966895 (+ 78us) server_negotiation.cc:407] Connection header received
0811 02:37:20.966977 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:20.966978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:20.966988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:20.967054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:20.967627 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.968134 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.968958 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.969201 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.969309 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.972121 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:20.972123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:20.972128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:20.972135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:20.973763 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.974503 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.974505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.974505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.974580 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:20.974847 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:20.974848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:20.974848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:20.974987 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:20.975048 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:20.975325 (+ 277us) server_negotiation.cc:299] Negotiation successful
0811 02:37:20.975472 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":33}
I20260811 02:37:20.975611 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:20.966642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44215 (local address 127.0.0.1:59904)
0811 02:37:20.966822 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:20.966833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:20.966910 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:20.967109 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:20.967110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:20.967470 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:20.967519 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.967522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.968238 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.968239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.968798 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:20.968800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:20.969336 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:20.969337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:20.969397 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:20.970062 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:20.970067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:20.972004 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:20.973909 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.973959 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.973964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.974347 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.974683 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:20.974684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:20.974685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:20.974766 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:20.975087 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:20.975096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:20.975171 (+ 75us) client_negotiation.cc:769] Sending connection context
0811 02:37:20.975248 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 02:37:20.975403 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260811 02:37:20.977965 5771 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59904:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 02:37:20.978163 5771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:20.979691 5991 tablet_service.cc:1467] Processing CreateTablet for tablet 1d043a81be204986859fe91ce9b3426a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bab42879007b43e68ea9c1d6744aec28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:20.979802 6222 tablet_service.cc:1467] Processing CreateTablet for tablet 1d043a81be204986859fe91ce9b3426a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bab42879007b43e68ea9c1d6744aec28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:20.979921 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d043a81be204986859fe91ce9b3426a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.979971 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d043a81be204986859fe91ce9b3426a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.981560 6623 tablet_bootstrap.cc:492] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: Bootstrap starting.
I20260811 02:37:20.981813 6625 tablet_bootstrap.cc:492] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: Bootstrap starting.
I20260811 02:37:20.981808 6445 tablet_service.cc:1467] Processing CreateTablet for tablet 1d043a81be204986859fe91ce9b3426a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bab42879007b43e68ea9c1d6744aec28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:20.981997 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d043a81be204986859fe91ce9b3426a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:20.982120 6623 tablet_bootstrap.cc:654] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.982582 6625 tablet_bootstrap.cc:654] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.983042 6623 tablet_bootstrap.cc:492] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: No bootstrap required, opened a new log
I20260811 02:37:20.983121 6623 ts_tablet_manager.cc:1397] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:20.983290 6623 raft_consensus.cc:348] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.983400 6623 raft_consensus.cc:374] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.983407 6625 tablet_bootstrap.cc:492] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: No bootstrap required, opened a new log
I20260811 02:37:20.983425 6623 raft_consensus.cc:729] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.983454 6625 ts_tablet_manager.cc:1397] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:20.983477 6623 consensus_queue.cc:260] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.983577 6623 ts_tablet_manager.cc:1428] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:20.983613 6625 raft_consensus.cc:348] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.983695 6625 raft_consensus.cc:374] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.983719 6625 raft_consensus.cc:729] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.983745 6631 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:20.983771 6625 consensus_queue.cc:260] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.983817 6620 tablet_bootstrap.cc:492] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: Bootstrap starting.
I20260811 02:37:20.983825 6631 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:20.983865 6625 ts_tablet_manager.cc:1428] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:20.984036 6634 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:20.984136 6634 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:20.984457 6620 tablet_bootstrap.cc:654] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:20.985306 6620 tablet_bootstrap.cc:492] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: No bootstrap required, opened a new log
I20260811 02:37:20.985357 6620 ts_tablet_manager.cc:1397] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:20.985518 6620 raft_consensus.cc:348] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.985577 6620 raft_consensus.cc:374] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:20.985595 6620 raft_consensus.cc:729] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d64d99accb4ab7b10587c7b093cff0, State: Initialized, Role: FOLLOWER
I20260811 02:37:20.985638 6620 consensus_queue.cc:260] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:20.985764 6620 ts_tablet_manager.cc:1428] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:20.985858 6628 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:20.985978 6628 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260811 02:37:20.990717 6103 tablet.cc:2369] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:37:21.100730 6583 tablet.cc:2369] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:21.209801 6634 raft_consensus.cc:482] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:21.209896 6634 raft_consensus.cc:504] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.210253 6634 leader_election.cc:288] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:21.214282 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.210523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:60215 (local address 127.2.251.2:38707)
0811 02:37:21.210598 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.210600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.210640 (+ 40us) server_negotiation.cc:407] Connection header received
0811 02:37:21.210727 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.210728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.210739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.210804 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:21.211034 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.211778 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.213281 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.213872 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.213947 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.213958 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.214154 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.214215 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 02:37:21.214279 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.210535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:44019 (local address 127.2.251.3:34959)
0811 02:37:21.210585 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.210587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.210638 (+ 51us) server_negotiation.cc:407] Connection header received
0811 02:37:21.210727 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.210728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.210755 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.210806 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:21.211009 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.211775 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.213281 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.213872 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.213947 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.213958 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.214154 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.214207 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 02:37:21.214287 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.210457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34959 (local address 127.2.251.1:44019)
0811 02:37:21.210578 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.210589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.210654 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.210823 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.210824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.210827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:21.210898 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.210902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.211875 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.211876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.213165 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.213170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.213971 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.213973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.214053 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.214057 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.214114 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.214172 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":75,"thread_start_us":61,"threads_started":1}
I20260811 02:37:21.214287 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.210449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38707 (local address 127.2.251.1:60215)
0811 02:37:21.210578 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.210589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.210654 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.210823 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.210824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.210827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:21.210928 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.210929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.211874 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.211876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.213176 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.213178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.213958 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.213960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.214053 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.214057 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.214114 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.214172 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":75,"thread_start_us":69,"threads_started":1}
I20260811 02:37:21.214533 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d043a81be204986859fe91ce9b3426a" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0" is_pre_election: true
I20260811 02:37:21.214618 6471 raft_consensus.cc:2454] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 0.
I20260811 02:37:21.214793 5921 leader_election.cc:302] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, 19d64d99accb4ab7b10587c7b093cff0; no voters:
I20260811 02:37:21.214787 6258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d043a81be204986859fe91ce9b3426a" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a354aa9503234e0bb8c9e456f11ba02d" is_pre_election: true
I20260811 02:37:21.214897 6258 raft_consensus.cc:2454] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 0.
I20260811 02:37:21.214931 6634 raft_consensus.cc:2781] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:21.214982 6634 raft_consensus.cc:482] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:21.215014 6634 raft_consensus.cc:3037] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:21.215706 6634 raft_consensus.cc:504] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.215835 6634 leader_election.cc:288] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 election: Requested vote from peers a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:21.215999 6258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d043a81be204986859fe91ce9b3426a" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a354aa9503234e0bb8c9e456f11ba02d"
I20260811 02:37:21.216005 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d043a81be204986859fe91ce9b3426a" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0"
I20260811 02:37:21.216076 6471 raft_consensus.cc:3037] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:21.216077 6258 raft_consensus.cc:3037] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:21.216831 6258 raft_consensus.cc:2454] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 1.
I20260811 02:37:21.216831 6471 raft_consensus.cc:2454] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 1.
I20260811 02:37:21.217037 5921 leader_election.cc:302] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, 19d64d99accb4ab7b10587c7b093cff0; no voters:
I20260811 02:37:21.217154 6634 raft_consensus.cc:2781] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:21.217296 6634 raft_consensus.cc:686] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 LEADER]: Becoming Leader. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Running, Role: LEADER
I20260811 02:37:21.217393 6634 consensus_queue.cc:237] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.217947 6654 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: TxnStatusTablet state changed. Reason: New leader 136b0fee7e7f4e81b9dcd8b6c494a401. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.218127 6654 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:21.218276 5771 catalog_manager.cc:5495] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:21.218394 6656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 02:37:21.218519 6656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 02:37:21.271576 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.268751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.2.251.1:45821)
0811 02:37:21.268807 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.268808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.268817 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:21.268879 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.268880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.268897 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.268958 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.269104 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.269588 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.270715 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.270921 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.270984 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.271221 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.271336 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.271410 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.271479 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.271508 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 02:37:21.271616 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.268658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.0.0.1:59574)
0811 02:37:21.268749 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.268759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.268805 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.268984 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.268985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.268986 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:21.269046 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.269049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.269663 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.269664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.270619 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.270622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.271002 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.271003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.271083 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.271102 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.271413 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.271421 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.271475 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.271530 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 02:37:21.273362 6471 raft_consensus.cc:1261] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Refusing update from remote peer 136b0fee7e7f4e81b9dcd8b6c494a401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:21.273368 6258 raft_consensus.cc:1261] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Refusing update from remote peer 136b0fee7e7f4e81b9dcd8b6c494a401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:21.273679 6634 consensus_queue.cc:1035] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:21.273841 6654 consensus_queue.cc:1035] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:21.275089 6628 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: TxnStatusTablet state changed. Reason: New leader 136b0fee7e7f4e81b9dcd8b6c494a401. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.275185 6628 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:21.276458 6631 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: TxnStatusTablet state changed. Reason: New leader 136b0fee7e7f4e81b9dcd8b6c494a401. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.276564 6631 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:21.277434 6660 mvcc.cc:205] Tried to move back new op lower bound from 7317159285852966912 to 7317159285626986496. Current Snapshot: MvccSnapshot[applied={T|T < 7317159285852966912}]
I20260811 02:37:21.277835 6634 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.277946 6634 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:21.278136 6661 mvcc.cc:205] Tried to move back new op lower bound from 7317159285852966912 to 7317159285626986496. Current Snapshot: MvccSnapshot[applied={T|T < 7317159285852966912}]
I20260811 02:37:21.278211 6628 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.278296 6628 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 19d64d99accb4ab7b10587c7b093cff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:21.277820 6669 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.278441 6669 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P 136b0fee7e7f4e81b9dcd8b6c494a401: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:21.279502 6631 tablet_replica.cc:439] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } } }
I20260811 02:37:21.279597 6631 tablet_replica.cc:442] T 1d043a81be204986859fe91ce9b3426a P a354aa9503234e0bb8c9e456f11ba02d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:21.280071 6663 mvcc.cc:205] Tried to move back new op lower bound from 7317159285852966912 to 7317159285626986496. Current Snapshot: MvccSnapshot[applied={T|T < 7317159285852966912 or (T in {7317159285852966912})}]
I20260811 02:37:21.283161 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.280545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.2.251.2:38707)
0811 02:37:21.280582 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.280584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.280606 (+ 22us) server_negotiation.cc:407] Connection header received
0811 02:37:21.280683 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.280684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.280703 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.280758 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.280869 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.281354 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.282317 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.282536 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.282584 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.282797 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.282930 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.282996 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.283075 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.283098 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:37:21.288359 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.285552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:45635 (local address 127.2.251.1:45821)
0811 02:37:21.285596 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.285598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.285800 (+ 202us) server_negotiation.cc:407] Connection header received
0811 02:37:21.285832 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.285847 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.285861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.285916 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.286084 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.286549 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.287625 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.287836 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.287897 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.288130 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.288164 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.288217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.288281 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.288301 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:21.288509 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.285481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.2.251.2:45635)
0811 02:37:21.285668 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.285681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.285778 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.285926 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.285927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.285928 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:21.285993 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.285994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.286663 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.286664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.287519 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.287522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.287917 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.287917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.287980 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.287998 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.288234 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.288236 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.288279 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.288324 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260811 02:37:21.291235 6471 raft_consensus.cc:1261] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Refusing update from remote peer a354aa9503234e0bb8c9e456f11ba02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:21.291306 6019 raft_consensus.cc:1261] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Refusing update from remote peer a354aa9503234e0bb8c9e456f11ba02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:21.291495 6637 consensus_queue.cc:1035] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:21.291628 6631 consensus_queue.cc:1035] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:21.304410 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.300837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:42051 (local address 127.2.251.2:38707)
0811 02:37:21.301054 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.301057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.301069 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:21.301104 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.301105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.301121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.301191 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.301397 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.302237 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.303515 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.303743 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.303818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.304084 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.304124 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.304186 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.304285 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.304310 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260811 02:37:21.304471 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.300720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38707 (local address 127.2.251.1:42051)
0811 02:37:21.300929 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.300942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.301001 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.301202 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.301204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.301207 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:21.301299 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.301301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.302349 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.302351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.303336 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.303340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.303848 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.303850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.303931 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.303953 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.304210 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.304212 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.304279 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.304340 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260811 02:37:21.315002 5775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:21.315126 5775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:21.315969 5775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Sending DeleteTablet for 3 replicas of tablet 6fd14302a4b145baa4f653aac8f9bf74
I20260811 02:37:21.316366 5991 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd14302a4b145baa4f653aac8f9bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:21 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59572
I20260811 02:37:21.316349 6445 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd14302a4b145baa4f653aac8f9bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:21 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260811 02:37:21.316613 6700 tablet_replica.cc:330] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: stopping tablet replica
I20260811 02:37:21.316738 6700 raft_consensus.cc:2229] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:21.316815 6700 raft_consensus.cc:2258] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:21.316816 6222 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd14302a4b145baa4f653aac8f9bf74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:21 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:37490
I20260811 02:37:21.317363 6700 ts_tablet_manager.cc:1905] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:21.317376 5775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:21.317533 5775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:21.318913 6445 tablet_service.cc:1467] Processing CreateTablet for tablet 56d7f2b6ec8d48a5a2ede33ae007204c (DEFAULT_TABLE table=test-table [id=6d1c153151b34ed3bf6abd50d49729fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:21.319032 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d7f2b6ec8d48a5a2ede33ae007204c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:21.319311 6700 ts_tablet_manager.cc:1918] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:21.319384 6700 log.cc:1196] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/6fd14302a4b145baa4f653aac8f9bf74
I20260811 02:37:21.319640 6700 ts_tablet_manager.cc:1939] T 6fd14302a4b145baa4f653aac8f9bf74 P 19d64d99accb4ab7b10587c7b093cff0: Deleting consensus metadata
I20260811 02:37:21.320657 5744 catalog_manager.cc:4841] TS 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959): tablet 6fd14302a4b145baa4f653aac8f9bf74 (table test-table [id=5e734fb81428436c814a2f1ec170ef90]) successfully deleted
I20260811 02:37:21.320819 6620 tablet_bootstrap.cc:492] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Bootstrap starting.
I20260811 02:37:21.321316 6620 tablet_bootstrap.cc:654] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:21.322214 6620 tablet_bootstrap.cc:492] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: No bootstrap required, opened a new log
I20260811 02:37:21.322319 6620 ts_tablet_manager.cc:1397] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:21.322512 6620 raft_consensus.cc:348] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.322602 6620 raft_consensus.cc:374] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:21.322623 6620 raft_consensus.cc:729] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d64d99accb4ab7b10587c7b093cff0, State: Initialized, Role: FOLLOWER
I20260811 02:37:21.322672 6620 consensus_queue.cc:260] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.322762 6620 ts_tablet_manager.cc:1428] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:21.323079 6222 tablet_service.cc:1467] Processing CreateTablet for tablet 56d7f2b6ec8d48a5a2ede33ae007204c (DEFAULT_TABLE table=test-table [id=6d1c153151b34ed3bf6abd50d49729fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:21.323238 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d7f2b6ec8d48a5a2ede33ae007204c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:21.324275 6623 tablet_bootstrap.cc:492] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Bootstrap starting.
I20260811 02:37:21.324774 6623 tablet_bootstrap.cc:654] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:21.325155 5991 tablet_service.cc:1467] Processing CreateTablet for tablet 56d7f2b6ec8d48a5a2ede33ae007204c (DEFAULT_TABLE table=test-table [id=6d1c153151b34ed3bf6abd50d49729fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:21.325284 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56d7f2b6ec8d48a5a2ede33ae007204c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:21.326531 6625 tablet_bootstrap.cc:492] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Bootstrap starting.
I20260811 02:37:21.326980 6625 tablet_bootstrap.cc:654] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:21.327482 6699 tablet_replica.cc:330] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: stopping tablet replica
I20260811 02:37:21.327622 6699 raft_consensus.cc:2229] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:21.327689 6699 raft_consensus.cc:2258] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:21.327855 6703 tablet_replica.cc:330] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: stopping tablet replica
I20260811 02:37:21.327970 6703 raft_consensus.cc:2229] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:21.328086 6703 raft_consensus.cc:2258] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:21.328341 6699 ts_tablet_manager.cc:1905] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:21.328567 6625 tablet_bootstrap.cc:492] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: No bootstrap required, opened a new log
I20260811 02:37:21.328619 6625 ts_tablet_manager.cc:1397] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:21.328778 6625 raft_consensus.cc:348] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.328857 6625 raft_consensus.cc:374] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:21.328878 6625 raft_consensus.cc:729] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Initialized, Role: FOLLOWER
I20260811 02:37:21.328917 6625 consensus_queue.cc:260] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.329002 6625 ts_tablet_manager.cc:1428] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:21.329835 6703 ts_tablet_manager.cc:1905] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:21.331969 6699 ts_tablet_manager.cc:1918] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:21.332056 6699 log.cc:1196] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/6fd14302a4b145baa4f653aac8f9bf74
I20260811 02:37:21.332333 6699 ts_tablet_manager.cc:1939] T 6fd14302a4b145baa4f653aac8f9bf74 P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting consensus metadata
I20260811 02:37:21.332593 6703 ts_tablet_manager.cc:1918] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:21.332726 6703 log.cc:1196] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/6fd14302a4b145baa4f653aac8f9bf74
I20260811 02:37:21.332824 5743 catalog_manager.cc:4841] TS 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821): tablet 6fd14302a4b145baa4f653aac8f9bf74 (table test-table [id=5e734fb81428436c814a2f1ec170ef90]) successfully deleted
I20260811 02:37:21.333179 6703 ts_tablet_manager.cc:1939] T 6fd14302a4b145baa4f653aac8f9bf74 P a354aa9503234e0bb8c9e456f11ba02d: Deleting consensus metadata
I20260811 02:37:21.333669 5743 catalog_manager.cc:4841] TS a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707): tablet 6fd14302a4b145baa4f653aac8f9bf74 (table test-table [id=5e734fb81428436c814a2f1ec170ef90]) successfully deleted
I20260811 02:37:21.334129 6623 tablet_bootstrap.cc:492] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: No bootstrap required, opened a new log
I20260811 02:37:21.334183 6623 ts_tablet_manager.cc:1397] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260811 02:37:21.334357 6623 raft_consensus.cc:348] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.334427 6623 raft_consensus.cc:374] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:21.334444 6623 raft_consensus.cc:729] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Initialized, Role: FOLLOWER
I20260811 02:37:21.334486 6623 consensus_queue.cc:260] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.334579 6623 ts_tablet_manager.cc:1428] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:21.345548 6628 raft_consensus.cc:482] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:21.345661 6628 raft_consensus.cc:504] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.346035 6628 leader_election.cc:288] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821), a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707)
I20260811 02:37:21.350250 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.346313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:38283 (local address 127.2.251.1:45821)
0811 02:37:21.346375 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.346378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.346529 (+ 151us) server_negotiation.cc:407] Connection header received
0811 02:37:21.346626 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.346627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.346637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.346666 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:21.346898 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.347672 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.349200 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.349812 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.349859 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.349872 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.350107 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.350169 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:37:21.350781 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.346217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.2.251.3:38283)
0811 02:37:21.346462 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.346474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.346551 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.346736 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.346737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.346739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:21.346822 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.346824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.347808 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.347809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.349070 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.349110 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.349947 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.349948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.350022 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.350027 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.350413 (+ 386us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.350474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":103}
I20260811 02:37:21.351011 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.346348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38707 (local address 127.2.251.3:57503)
0811 02:37:21.346477 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.346484 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.346529 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.346856 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.346857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.346859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:21.346904 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.346906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.347815 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.347816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.349070 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.349074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.350752 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.350753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.350830 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.350834 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.350901 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.350949 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"spinlock_wait_cycles":1280,"thread_start_us":69,"threads_started":1}
I20260811 02:37:21.351016 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.346671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:57503 (local address 127.2.251.2:38707)
0811 02:37:21.346716 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.346717 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.346725 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:21.346750 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.346750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.346758 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.346802 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:21.346969 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.347695 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.349284 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.349960 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.350013 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.350023 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.350912 (+ 889us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.350953 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:21.351460 6258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7f2b6ec8d48a5a2ede33ae007204c" candidate_uuid: "19d64d99accb4ab7b10587c7b093cff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a354aa9503234e0bb8c9e456f11ba02d" is_pre_election: true
I20260811 02:37:21.351480 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7f2b6ec8d48a5a2ede33ae007204c" candidate_uuid: "19d64d99accb4ab7b10587c7b093cff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" is_pre_election: true
I20260811 02:37:21.351560 6258 raft_consensus.cc:2454] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d64d99accb4ab7b10587c7b093cff0 in term 0.
I20260811 02:37:21.351560 6019 raft_consensus.cc:2454] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d64d99accb4ab7b10587c7b093cff0 in term 0.
I20260811 02:37:21.351930 6369 leader_election.cc:302] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d64d99accb4ab7b10587c7b093cff0, a354aa9503234e0bb8c9e456f11ba02d; no voters:
I20260811 02:37:21.352139 6628 raft_consensus.cc:2781] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:21.352177 6628 raft_consensus.cc:482] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:21.352197 6628 raft_consensus.cc:3037] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:21.352949 6628 raft_consensus.cc:504] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.353080 6628 leader_election.cc:288] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [CANDIDATE]: Term 1 election: Requested vote from peers 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821), a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707)
I20260811 02:37:21.353276 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7f2b6ec8d48a5a2ede33ae007204c" candidate_uuid: "19d64d99accb4ab7b10587c7b093cff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401"
I20260811 02:37:21.353310 6258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56d7f2b6ec8d48a5a2ede33ae007204c" candidate_uuid: "19d64d99accb4ab7b10587c7b093cff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a354aa9503234e0bb8c9e456f11ba02d"
I20260811 02:37:21.353359 6019 raft_consensus.cc:3037] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:21.353374 6258 raft_consensus.cc:3037] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:21.354372 6258 raft_consensus.cc:2454] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d64d99accb4ab7b10587c7b093cff0 in term 1.
I20260811 02:37:21.354422 6019 raft_consensus.cc:2454] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d64d99accb4ab7b10587c7b093cff0 in term 1.
I20260811 02:37:21.354573 6369 leader_election.cc:302] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, 19d64d99accb4ab7b10587c7b093cff0; no voters:
I20260811 02:37:21.354682 6628 raft_consensus.cc:2781] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:21.354800 6628 raft_consensus.cc:686] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 1 LEADER]: Becoming Leader. State: Replica: 19d64d99accb4ab7b10587c7b093cff0, State: Running, Role: LEADER
I20260811 02:37:21.354880 6628 consensus_queue.cc:237] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:21.355636 5775 catalog_manager.cc:5495] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3). New cstate: current_term: 1 leader_uuid: "19d64d99accb4ab7b10587c7b093cff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } health_report { overall_health: HEALTHY } } }
I20260811 02:37:21.365150 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.362562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.2.251.3:34959)
0811 02:37:21.362756 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.362758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.362771 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:21.362850 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.362851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.362875 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.362948 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.363082 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.363634 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.364370 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.364566 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.364620 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.364774 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.364930 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.364999 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.365045 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.365066 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260811 02:37:21.370311 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.367871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:60337 (local address 127.2.251.1:45821)
0811 02:37:21.367926 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.367928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.368040 (+ 112us) server_negotiation.cc:407] Connection header received
0811 02:37:21.368095 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.368096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.368107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.368151 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.368272 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.368758 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.369560 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.369819 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.369877 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.370057 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.370094 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.370139 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.370197 (+ 58us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.370221 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:37:21.370324 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.367751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.2.251.3:60337)
0811 02:37:21.367962 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.367975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.368042 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.368153 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.368154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.368155 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:21.368218 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.368220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.368839 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.368841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.369467 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.369469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.369884 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.369885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.369962 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.369979 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.370147 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.370149 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.370186 (+ 37us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.370238 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260811 02:37:21.372907 6019 raft_consensus.cc:1261] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Refusing update from remote peer 19d64d99accb4ab7b10587c7b093cff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:21.373293 6711 consensus_queue.cc:1035] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:21.373560 6258 raft_consensus.cc:1261] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Refusing update from remote peer 19d64d99accb4ab7b10587c7b093cff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:21.374538 6628 consensus_queue.cc:1035] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:21.386415 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.383582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:53171 (local address 127.2.251.3:34959)
0811 02:37:21.383639 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:21.383641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:21.383714 (+ 73us) server_negotiation.cc:407] Connection header received
0811 02:37:21.383762 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:21.383763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:21.383776 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:21.383833 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:21.384167 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.384766 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.385779 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.385984 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.386040 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.386180 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.386216 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.386261 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:21.386315 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 02:37:21.386337 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:37:21.386437 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:21.383545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34959 (local address 127.2.251.1:53171)
0811 02:37:21.383645 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:21.383656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:21.383708 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:21.383932 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:21.383934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:21.383936 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:21.384073 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.384077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.384845 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.384846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.385645 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:21.385648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:21.386051 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:21.386052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:21.386109 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:21.386117 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:21.386271 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:21.386272 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 02:37:21.386312 (+ 40us) client_negotiation.cc:240] Negotiation successful
0811 02:37:21.386371 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
W20260811 02:37:21.386898 6661 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 02:37:22.912150 5775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:22.912278 5775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:22.913343 5775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Sending DeleteTablet for 3 replicas of tablet 56d7f2b6ec8d48a5a2ede33ae007204c
I20260811 02:37:22.913630 6445 tablet_service.cc:1514] Processing DeleteTablet for tablet 56d7f2b6ec8d48a5a2ede33ae007204c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:22 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260811 02:37:22.913703 5991 tablet_service.cc:1514] Processing DeleteTablet for tablet 56d7f2b6ec8d48a5a2ede33ae007204c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:22 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59572
I20260811 02:37:22.913861 6222 tablet_service.cc:1514] Processing DeleteTablet for tablet 56d7f2b6ec8d48a5a2ede33ae007204c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:22 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:37490
I20260811 02:37:22.914131 6766 tablet_replica.cc:330] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: stopping tablet replica
I20260811 02:37:22.914252 6766 raft_consensus.cc:2229] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:22.914348 6766 raft_consensus.cc:2258] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:22.914649 6766 ts_tablet_manager.cc:1905] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:22.916509 6766 ts_tablet_manager.cc:1918] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 02:37:22.916581 6766 log.cc:1196] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/56d7f2b6ec8d48a5a2ede33ae007204c
I20260811 02:37:22.916792 6766 ts_tablet_manager.cc:1939] T 56d7f2b6ec8d48a5a2ede33ae007204c P a354aa9503234e0bb8c9e456f11ba02d: Deleting consensus metadata
I20260811 02:37:22.917402 5743 catalog_manager.cc:4841] TS a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707): tablet 56d7f2b6ec8d48a5a2ede33ae007204c (table test-table [id=6d1c153151b34ed3bf6abd50d49729fa]) successfully deleted
I20260811 02:37:22.917853 6765 tablet_replica.cc:330] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: stopping tablet replica
I20260811 02:37:22.917939 6765 raft_consensus.cc:2229] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:22.917981 6765 raft_consensus.cc:2258] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:22.918126 6764 tablet_replica.cc:330] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: stopping tablet replica
I20260811 02:37:22.918200 6764 raft_consensus.cc:2229] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:22.918486 6764 raft_consensus.cc:2258] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:22.918443 6765 ts_tablet_manager.cc:1905] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:22.918833 6764 ts_tablet_manager.cc:1905] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:22.921494 6765 ts_tablet_manager.cc:1918] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 02:37:22.921802 6765 log.cc:1196] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/56d7f2b6ec8d48a5a2ede33ae007204c
I20260811 02:37:22.922106 6765 ts_tablet_manager.cc:1939] T 56d7f2b6ec8d48a5a2ede33ae007204c P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting consensus metadata
I20260811 02:37:22.922103 6764 ts_tablet_manager.cc:1918] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 02:37:22.922169 6764 log.cc:1196] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/56d7f2b6ec8d48a5a2ede33ae007204c
I20260811 02:37:22.922405 6764 ts_tablet_manager.cc:1939] T 56d7f2b6ec8d48a5a2ede33ae007204c P 19d64d99accb4ab7b10587c7b093cff0: Deleting consensus metadata
I20260811 02:37:22.922891 5743 catalog_manager.cc:4841] TS 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821): tablet 56d7f2b6ec8d48a5a2ede33ae007204c (table test-table [id=6d1c153151b34ed3bf6abd50d49729fa]) successfully deleted
I20260811 02:37:22.923913 5744 catalog_manager.cc:4841] TS 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959): tablet 56d7f2b6ec8d48a5a2ede33ae007204c (table test-table [id=6d1c153151b34ed3bf6abd50d49729fa]) successfully deleted
Aug 11 02:37:22 dist-test-slave-cvwc krb5kdc[5694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 02:37:22 dist-test-slave-cvwc krb5kdc[5694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415842, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:22.956153 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:22.941247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.2.251.62:44215)
0811 02:37:22.941449 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:22.941451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:22.941464 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:22.941578 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:22.941579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:22.941596 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:22.941669 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:22.943177 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.944114 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.945032 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.945482 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.945597 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:22.953290 (+ 7693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:22.953292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:22.953293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:22.953302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:22.954880 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:22.955299 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:22.955301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:22.955301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:22.955377 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:22.955613 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:22.955614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:22.955614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:22.955743 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:22.955824 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:22.955985 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 02:37:22.956064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260811 02:37:22.958705 5775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:22.958963 5775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:22.961372 6445 tablet_service.cc:1467] Processing CreateTablet for tablet 2f46450ed97040b8b6d62c5b237fb835 (DEFAULT_TABLE table=test-table [id=62a874adb4bd4f1087059190af526f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:22.961455 6222 tablet_service.cc:1467] Processing CreateTablet for tablet 2f46450ed97040b8b6d62c5b237fb835 (DEFAULT_TABLE table=test-table [id=62a874adb4bd4f1087059190af526f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:22.961455 5991 tablet_service.cc:1467] Processing CreateTablet for tablet 2f46450ed97040b8b6d62c5b237fb835 (DEFAULT_TABLE table=test-table [id=62a874adb4bd4f1087059190af526f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:22.961556 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f46450ed97040b8b6d62c5b237fb835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:22.961798 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f46450ed97040b8b6d62c5b237fb835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:22.961813 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f46450ed97040b8b6d62c5b237fb835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:22.963248 6781 tablet_bootstrap.cc:492] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Bootstrap starting.
I20260811 02:37:22.963238 6780 tablet_bootstrap.cc:492] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Bootstrap starting.
I20260811 02:37:22.963375 6779 tablet_bootstrap.cc:492] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Bootstrap starting.
I20260811 02:37:22.963666 6780 tablet_bootstrap.cc:654] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:22.963845 6779 tablet_bootstrap.cc:654] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:22.963857 6781 tablet_bootstrap.cc:654] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:22.964489 6781 tablet_bootstrap.cc:492] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: No bootstrap required, opened a new log
I20260811 02:37:22.964489 6780 tablet_bootstrap.cc:492] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: No bootstrap required, opened a new log
I20260811 02:37:22.964550 6780 ts_tablet_manager.cc:1397] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:22.964550 6781 ts_tablet_manager.cc:1397] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:22.964596 6779 tablet_bootstrap.cc:492] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: No bootstrap required, opened a new log
I20260811 02:37:22.964645 6779 ts_tablet_manager.cc:1397] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:22.964681 6781 raft_consensus.cc:348] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.964679 6780 raft_consensus.cc:348] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.964761 6780 raft_consensus.cc:374] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:22.964761 6781 raft_consensus.cc:374] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:22.964787 6781 raft_consensus.cc:729] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Initialized, Role: FOLLOWER
I20260811 02:37:22.964787 6780 raft_consensus.cc:729] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Initialized, Role: FOLLOWER
I20260811 02:37:22.964771 6779 raft_consensus.cc:348] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.964847 6779 raft_consensus.cc:374] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:22.964836 6781 consensus_queue.cc:260] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.964867 6779 raft_consensus.cc:729] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d64d99accb4ab7b10587c7b093cff0, State: Initialized, Role: FOLLOWER
I20260811 02:37:22.964862 6780 consensus_queue.cc:260] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.964941 6781 ts_tablet_manager.cc:1428] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:22.964933 6779 consensus_queue.cc:260] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.965075 6779 ts_tablet_manager.cc:1428] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:22.965091 6780 ts_tablet_manager.cc:1428] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:22.971658 6729 raft_consensus.cc:482] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:22.971761 6729 raft_consensus.cc:504] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.971944 6729 leader_election.cc:288] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:22.972201 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46450ed97040b8b6d62c5b237fb835" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0" is_pre_election: true
I20260811 02:37:22.972213 6258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46450ed97040b8b6d62c5b237fb835" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a354aa9503234e0bb8c9e456f11ba02d" is_pre_election: true
I20260811 02:37:22.972323 6258 raft_consensus.cc:2454] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 0.
I20260811 02:37:22.972343 6471 raft_consensus.cc:2454] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 0.
I20260811 02:37:22.972529 5921 leader_election.cc:302] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, 19d64d99accb4ab7b10587c7b093cff0; no voters:
I20260811 02:37:22.972649 6729 raft_consensus.cc:2781] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:22.972700 6729 raft_consensus.cc:482] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:22.972729 6729 raft_consensus.cc:3037] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:22.973479 6729 raft_consensus.cc:504] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.973613 6729 leader_election.cc:288] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 election: Requested vote from peers a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:22.973831 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46450ed97040b8b6d62c5b237fb835" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0"
I20260811 02:37:22.973927 6471 raft_consensus.cc:3037] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:22.973912 6258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46450ed97040b8b6d62c5b237fb835" candidate_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a354aa9503234e0bb8c9e456f11ba02d"
I20260811 02:37:22.973994 6258 raft_consensus.cc:3037] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:22.974763 6258 raft_consensus.cc:2454] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 1.
I20260811 02:37:22.974763 6471 raft_consensus.cc:2454] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136b0fee7e7f4e81b9dcd8b6c494a401 in term 1.
I20260811 02:37:22.974943 5921 leader_election.cc:302] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, 19d64d99accb4ab7b10587c7b093cff0; no voters:
I20260811 02:37:22.975064 6729 raft_consensus.cc:2781] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:22.975137 6729 raft_consensus.cc:686] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 LEADER]: Becoming Leader. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Running, Role: LEADER
I20260811 02:37:22.975190 6729 consensus_queue.cc:237] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:22.975842 5771 catalog_manager.cc:5495] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:22.988976 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:22.986348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.2.251.1:45821)
0811 02:37:22.986580 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:22.986582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:22.986590 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:22.986616 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:22.986616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:22.986629 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:22.986697 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:22.986840 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.987367 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.988169 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.988395 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.988470 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:22.988635 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:22.988671 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:22.988714 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:22.988776 (+ 62us) server_negotiation.cc:299] Negotiation successful
0811 02:37:22.988801 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":161,"threads_started":1}
I20260811 02:37:22.994620 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:22.991856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:56439 (local address 127.2.251.1:45821)
0811 02:37:22.991894 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:22.991896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:22.991994 (+ 98us) server_negotiation.cc:407] Connection header received
0811 02:37:22.992028 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:22.992029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:22.992039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:22.992086 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:22.992257 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.992753 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.993944 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.994197 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.994262 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:22.994410 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:22.994442 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:22.994478 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:22.994528 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 02:37:22.994548 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 02:37:22.994632 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:22.991720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.2.251.1:56439)
0811 02:37:22.991884 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:22.991896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:22.991966 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:22.992105 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:22.992105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:22.992108 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:22.992163 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:22.992166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.992825 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.992825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:22.993859 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:22.993862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:22.994285 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:22.994286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:22.994344 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:22.994351 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:22.994490 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:22.994492 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:37:22.994521 (+ 29us) client_negotiation.cc:240] Negotiation successful
0811 02:37:22.994564 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260811 02:37:22.996682 6471 raft_consensus.cc:1261] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Refusing update from remote peer 136b0fee7e7f4e81b9dcd8b6c494a401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:22.996752 6258 raft_consensus.cc:1261] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Refusing update from remote peer 136b0fee7e7f4e81b9dcd8b6c494a401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:22.996984 6739 consensus_queue.cc:1035] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:22.997133 6654 consensus_queue.cc:1035] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:23.010551 5770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:23.010679 5770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:23.011608 5770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Sending DeleteTablet for 3 replicas of tablet 2f46450ed97040b8b6d62c5b237fb835
I20260811 02:37:23.011926 5990 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f46450ed97040b8b6d62c5b237fb835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:23 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59572
I20260811 02:37:23.012053 6445 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f46450ed97040b8b6d62c5b237fb835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:23 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260811 02:37:23.012060 6222 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f46450ed97040b8b6d62c5b237fb835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:23 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:37490
I20260811 02:37:23.012171 6765 tablet_replica.cc:330] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: stopping tablet replica
I20260811 02:37:23.012239 6765 raft_consensus.cc:2229] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:23.012349 6765 raft_consensus.cc:2258] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:23.012308 5770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 02:37:23.012440 6764 tablet_replica.cc:330] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: stopping tablet replica
W20260811 02:37:23.012494 5770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:23.012499 6764 raft_consensus.cc:2229] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:23.012537 6764 raft_consensus.cc:2258] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:23.012614 6766 tablet_replica.cc:330] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: stopping tablet replica
I20260811 02:37:23.012683 6766 raft_consensus.cc:2229] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:23.012732 6766 raft_consensus.cc:2258] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:23.013093 6765 ts_tablet_manager.cc:1905] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:23.013419 6764 ts_tablet_manager.cc:1905] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:23.013895 6445 tablet_service.cc:1467] Processing CreateTablet for tablet c89cff08a79e423cbffd909596789a9f (DEFAULT_TABLE table=test-table [id=93619b4d676f4b5fb386991f94ef2d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:23.014008 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89cff08a79e423cbffd909596789a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:23.015062 6765 ts_tablet_manager.cc:1918] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:23.015096 6764 ts_tablet_manager.cc:1918] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:23.015122 6765 log.cc:1196] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/2f46450ed97040b8b6d62c5b237fb835
I20260811 02:37:23.015151 6764 log.cc:1196] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/2f46450ed97040b8b6d62c5b237fb835
I20260811 02:37:23.015177 6766 ts_tablet_manager.cc:1905] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:23.015349 6765 ts_tablet_manager.cc:1939] T 2f46450ed97040b8b6d62c5b237fb835 P 136b0fee7e7f4e81b9dcd8b6c494a401: Deleting consensus metadata
I20260811 02:37:23.015369 6764 ts_tablet_manager.cc:1939] T 2f46450ed97040b8b6d62c5b237fb835 P 19d64d99accb4ab7b10587c7b093cff0: Deleting consensus metadata
I20260811 02:37:23.015627 6779 tablet_bootstrap.cc:492] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Bootstrap starting.
I20260811 02:37:23.015658 5990 tablet_service.cc:1467] Processing CreateTablet for tablet c89cff08a79e423cbffd909596789a9f (DEFAULT_TABLE table=test-table [id=93619b4d676f4b5fb386991f94ef2d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:23.015731 5743 catalog_manager.cc:4841] TS 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821): tablet 2f46450ed97040b8b6d62c5b237fb835 (table test-table [id=62a874adb4bd4f1087059190af526f84]) successfully deleted
I20260811 02:37:23.015767 5990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89cff08a79e423cbffd909596789a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:23.015806 5744 catalog_manager.cc:4841] TS 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959): tablet 2f46450ed97040b8b6d62c5b237fb835 (table test-table [id=62a874adb4bd4f1087059190af526f84]) successfully deleted
I20260811 02:37:23.015990 6779 tablet_bootstrap.cc:654] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:23.017469 6781 tablet_bootstrap.cc:492] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401: Bootstrap starting.
I20260811 02:37:23.017592 6766 ts_tablet_manager.cc:1918] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:23.017606 6779 tablet_bootstrap.cc:492] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: No bootstrap required, opened a new log
I20260811 02:37:23.017648 6779 ts_tablet_manager.cc:1397] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:23.017648 6766 log.cc:1196] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/2f46450ed97040b8b6d62c5b237fb835
I20260811 02:37:23.017652 6222 tablet_service.cc:1467] Processing CreateTablet for tablet c89cff08a79e423cbffd909596789a9f (DEFAULT_TABLE table=test-table [id=93619b4d676f4b5fb386991f94ef2d54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:23.017807 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c89cff08a79e423cbffd909596789a9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:23.017848 6781 tablet_bootstrap.cc:654] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:23.017863 6779 raft_consensus.cc:348] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.017930 6766 ts_tablet_manager.cc:1939] T 2f46450ed97040b8b6d62c5b237fb835 P a354aa9503234e0bb8c9e456f11ba02d: Deleting consensus metadata
I20260811 02:37:23.017939 6779 raft_consensus.cc:374] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:23.017961 6779 raft_consensus.cc:729] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d64d99accb4ab7b10587c7b093cff0, State: Initialized, Role: FOLLOWER
I20260811 02:37:23.018008 6779 consensus_queue.cc:260] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.018143 6779 ts_tablet_manager.cc:1428] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:23.018321 5743 catalog_manager.cc:4841] TS a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707): tablet 2f46450ed97040b8b6d62c5b237fb835 (table test-table [id=62a874adb4bd4f1087059190af526f84]) successfully deleted
I20260811 02:37:23.019054 6780 tablet_bootstrap.cc:492] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Bootstrap starting.
I20260811 02:37:23.019377 6780 tablet_bootstrap.cc:654] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:23.019426 6781 tablet_bootstrap.cc:492] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401: No bootstrap required, opened a new log
I20260811 02:37:23.019465 6781 ts_tablet_manager.cc:1397] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:23.019568 6781 raft_consensus.cc:348] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.019636 6781 raft_consensus.cc:374] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:23.019655 6781 raft_consensus.cc:729] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136b0fee7e7f4e81b9dcd8b6c494a401, State: Initialized, Role: FOLLOWER
I20260811 02:37:23.019699 6781 consensus_queue.cc:260] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.019817 6781 ts_tablet_manager.cc:1428] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:23.020412 6780 tablet_bootstrap.cc:492] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: No bootstrap required, opened a new log
I20260811 02:37:23.020470 6780 ts_tablet_manager.cc:1397] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:23.020584 6780 raft_consensus.cc:348] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.020648 6780 raft_consensus.cc:374] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:23.020666 6780 raft_consensus.cc:729] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Initialized, Role: FOLLOWER
I20260811 02:37:23.020713 6780 consensus_queue.cc:260] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.020802 6780 ts_tablet_manager.cc:1428] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:23.308012 6789 raft_consensus.cc:482] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:23.308195 6789 raft_consensus.cc:504] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.308619 6789 leader_election.cc:288] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:23.308915 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cff08a79e423cbffd909596789a9f" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" is_pre_election: true
I20260811 02:37:23.308914 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cff08a79e423cbffd909596789a9f" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0" is_pre_election: true
I20260811 02:37:23.309042 6019 raft_consensus.cc:2454] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 0.
I20260811 02:37:23.309042 6471 raft_consensus.cc:2454] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 0.
I20260811 02:37:23.309217 6141 leader_election.cc:302] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d64d99accb4ab7b10587c7b093cff0, a354aa9503234e0bb8c9e456f11ba02d; no voters:
I20260811 02:37:23.309391 6789 raft_consensus.cc:2781] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:23.309448 6789 raft_consensus.cc:482] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:23.309469 6789 raft_consensus.cc:3037] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:23.310329 6789 raft_consensus.cc:504] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.310483 6789 leader_election.cc:288] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 election: Requested vote from peers 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821), 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959)
I20260811 02:37:23.310660 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cff08a79e423cbffd909596789a9f" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d64d99accb4ab7b10587c7b093cff0"
I20260811 02:37:23.310675 6019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c89cff08a79e423cbffd909596789a9f" candidate_uuid: "a354aa9503234e0bb8c9e456f11ba02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401"
I20260811 02:37:23.310742 6471 raft_consensus.cc:3037] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:23.310742 6019 raft_consensus.cc:3037] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:23.311396 6019 raft_consensus.cc:2454] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 1.
I20260811 02:37:23.311406 6471 raft_consensus.cc:2454] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a354aa9503234e0bb8c9e456f11ba02d in term 1.
I20260811 02:37:23.311587 6140 leader_election.cc:302] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136b0fee7e7f4e81b9dcd8b6c494a401, a354aa9503234e0bb8c9e456f11ba02d; no voters:
I20260811 02:37:23.311694 6789 raft_consensus.cc:2781] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:23.311805 6789 raft_consensus.cc:686] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 1 LEADER]: Becoming Leader. State: Replica: a354aa9503234e0bb8c9e456f11ba02d, State: Running, Role: LEADER
I20260811 02:37:23.311875 6789 consensus_queue.cc:237] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } }
I20260811 02:37:23.312577 5771 catalog_manager.cc:5495] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d reported cstate change: term changed from 0 to 1, leader changed from <none> to a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2). New cstate: current_term: 1 leader_uuid: "a354aa9503234e0bb8c9e456f11ba02d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a354aa9503234e0bb8c9e456f11ba02d" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:23.382704 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:23.378882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.2.251.2:38707)
0811 02:37:23.379267 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:23.379269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:23.379280 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:23.379318 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:23.379319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:23.379332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:23.379411 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:23.379769 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:23.380400 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:23.381413 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:23.381843 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:23.381921 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:23.382286 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:23.382335 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:23.382422 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:23.382569 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 02:37:23.382614 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":274,"threads_started":1}
I20260811 02:37:23.386286 6019 raft_consensus.cc:1261] T c89cff08a79e423cbffd909596789a9f P 136b0fee7e7f4e81b9dcd8b6c494a401 [term 1 FOLLOWER]: Refusing update from remote peer a354aa9503234e0bb8c9e456f11ba02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:23.386734 6471 raft_consensus.cc:1261] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Refusing update from remote peer a354aa9503234e0bb8c9e456f11ba02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:23.386715 6805 consensus_queue.cc:1035] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "136b0fee7e7f4e81b9dcd8b6c494a401" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:23.387081 6805 consensus_queue.cc:1035] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19d64d99accb4ab7b10587c7b093cff0" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 02:37:23.393090 6735 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 02:37:24.919591 5771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:24.919759 5771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:24.921294 5771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c14e15b1bbe40de8ccb5bb32ac890ff: Sending DeleteTablet for 3 replicas of tablet c89cff08a79e423cbffd909596789a9f
I20260811 02:37:24.921995 5990 tablet_service.cc:1514] Processing DeleteTablet for tablet c89cff08a79e423cbffd909596789a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:24 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59572
I20260811 02:37:24.922107 6222 tablet_service.cc:1514] Processing DeleteTablet for tablet c89cff08a79e423cbffd909596789a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:24 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:37490
I20260811 02:37:24.922335 6445 tablet_service.cc:1514] Processing DeleteTablet for tablet c89cff08a79e423cbffd909596789a9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:24 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260811 02:37:24.922501 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5889
I20260811 02:37:24.939877 6861 tablet_replica.cc:330] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: stopping tablet replica
I20260811 02:37:24.940052 6858 tablet_replica.cc:330] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: stopping tablet replica
I20260811 02:37:24.940197 6861 raft_consensus.cc:2229] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:24.940373 6861 raft_consensus.cc:2258] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:24.944434 6858 raft_consensus.cc:2229] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:24.944646 6861 ts_tablet_manager.cc:1905] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:24.944832 6858 raft_consensus.cc:2258] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:24.948132 6861 ts_tablet_manager.cc:1918] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 02:37:24.948352 6861 log.cc:1196] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/c89cff08a79e423cbffd909596789a9f
I20260811 02:37:24.948402 6858 ts_tablet_manager.cc:1905] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:24.948812 6861 ts_tablet_manager.cc:1939] T c89cff08a79e423cbffd909596789a9f P 19d64d99accb4ab7b10587c7b093cff0: Deleting consensus metadata
I20260811 02:37:24.949764 5744 catalog_manager.cc:4841] TS 19d64d99accb4ab7b10587c7b093cff0 (127.2.251.3:34959): tablet c89cff08a79e423cbffd909596789a9f (table test-table [id=93619b4d676f4b5fb386991f94ef2d54]) successfully deleted
I20260811 02:37:24.951063 6858 ts_tablet_manager.cc:1918] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 02:37:24.951156 6858 log.cc:1196] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/c89cff08a79e423cbffd909596789a9f
I20260811 02:37:24.951354 6858 ts_tablet_manager.cc:1939] T c89cff08a79e423cbffd909596789a9f P a354aa9503234e0bb8c9e456f11ba02d: Deleting consensus metadata
I20260811 02:37:24.951900 5743 catalog_manager.cc:4841] TS a354aa9503234e0bb8c9e456f11ba02d (127.2.251.2:38707): tablet c89cff08a79e423cbffd909596789a9f (table test-table [id=93619b4d676f4b5fb386991f94ef2d54]) successfully deleted
W20260811 02:37:24.958811 5743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:45821 (error 108)
I20260811 02:37:24.959180 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6116
I20260811 02:37:24.973235 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:24.972796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45821 (local address 127.0.0.1:39634)
0811 02:37:24.973078 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:24.973171 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:45821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":129,"threads_started":1}
W20260811 02:37:24.973402 5743 catalog_manager.cc:4584] TS 136b0fee7e7f4e81b9dcd8b6c494a401 (127.2.251.1:45821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c89cff08a79e423cbffd909596789a9f: Network error: Client connection negotiation failed: client connection to 127.2.251.1:45821: connect: Connection refused (error 111)
I20260811 02:37:24.980607 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6344
I20260811 02:37:25.000277 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5716
2026-08-11T02:37:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6239 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:25 dist-test-slave-cvwc krb5kdc[6866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:25 dist-test-slave-cvwc krb5kdc[6866](info): set up 2 sockets
Aug 11 02:37:25 dist-test-slave-cvwc krb5kdc[6866](info): commencing operation
krb5kdc: starting...
W20260811 02:37:26.262272 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:26.401116 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39013
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:26.495098 6885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:26.495362 6885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:26.495395 6885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:26.496694 6885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:26.496760 6885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:26.496780 6885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:26.496797 6885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:26.496811 6885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:26.498549 6885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:26.499094 6885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:26.499404 6885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:26.505049 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:26.505231 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:26.505049 6898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:26.505406 6885 server_base.cc:1034] running on GCE node
I20260811 02:37:26.505873 6885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:26.506302 6885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:26.507511 6885 hybrid_clock.cc:648] HybridClock initialized: now 1786415846507477 us; error 46 us; skew 500 ppm
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:26.510808 6885 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:26.511346 6885 webserver.cc:460] Webserver started at http://127.2.251.62:41561/ using document root <none> and password file <none>
I20260811 02:37:26.511575 6885 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:26.511623 6885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:26.511757 6885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:26.512925 6885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "2d01139748444826b0ab34b1e13a21f5"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "291e2c35fc31da834fbef72a30c67a19"
server_key_iv: "7b305e2d95eba99f59b87554777f9d20"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.513289 6885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "2d01139748444826b0ab34b1e13a21f5"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "291e2c35fc31da834fbef72a30c67a19"
server_key_iv: "7b305e2d95eba99f59b87554777f9d20"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.515048 6885 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 02:37:26.516060 6907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.516327 6885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:26.516422 6885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "2d01139748444826b0ab34b1e13a21f5"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "291e2c35fc31da834fbef72a30c67a19"
server_key_iv: "7b305e2d95eba99f59b87554777f9d20"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.516503 6885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:26.544576 6885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:26.547463 6885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:26.547605 6885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:26.551669 6885 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39013
I20260811 02:37:26.551829 6983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39013 every 8 connection(s)
I20260811 02:37:26.552085 6885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:26.553048 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:26.555552 6985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5: Bootstrap starting.
I20260811 02:37:26.556190 6985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:26.556489 6985 log.cc:824] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:26.557528 6985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5: No bootstrap required, opened a new log
I20260811 02:37:26.559116 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6885
I20260811 02:37:26.559249 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:26.559340 6985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d01139748444826b0ab34b1e13a21f5" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39013 } }
I20260811 02:37:26.559476 3052 external_mini_cluster.cc:1428] Setting key 0334061fd61bf0a96594dd001aec5033
I20260811 02:37:26.559509 6985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:26.559551 6985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d01139748444826b0ab34b1e13a21f5, State: Initialized, Role: FOLLOWER
I20260811 02:37:26.559705 6985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d01139748444826b0ab34b1e13a21f5" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39013 } }
I20260811 02:37:26.559790 6985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:26.559825 6985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:26.559875 6985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:26.560868 6985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d01139748444826b0ab34b1e13a21f5" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39013 } }
I20260811 02:37:26.560986 6985 leader_election.cc:302] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d01139748444826b0ab34b1e13a21f5; no voters:
I20260811 02:37:26.561172 6985 leader_election.cc:288] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:26.561240 6993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:26.561447 6985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:26.561479 6993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [term 1 LEADER]: Becoming Leader. State: Replica: 2d01139748444826b0ab34b1e13a21f5, State: Running, Role: LEADER
I20260811 02:37:26.562796 6993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d01139748444826b0ab34b1e13a21f5" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39013 } }
I20260811 02:37:26.563441 6994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d01139748444826b0ab34b1e13a21f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d01139748444826b0ab34b1e13a21f5" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39013 } } }
I20260811 02:37:26.563532 6994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:26.563939 7016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:26.563997 6995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d01139748444826b0ab34b1e13a21f5. Latest consensus state: current_term: 1 leader_uuid: "2d01139748444826b0ab34b1e13a21f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d01139748444826b0ab34b1e13a21f5" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39013 } } }
I20260811 02:37:26.564064 6995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:26.564455 7016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:26.566090 7016 catalog_manager.cc:1347] Generated new cluster ID: ca8e3db7c3794163bfe4f08e800674f6
I20260811 02:37:26.566152 7016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:26.571780 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:26.560065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.2.251.62:39013)
0811 02:37:26.560318 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:26.560330 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:26.560532 (+ 202us) server_negotiation.cc:407] Connection header received
0811 02:37:26.560746 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:26.560753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:26.560805 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:26.560944 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:26.561833 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.563174 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.564267 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.564740 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.564845 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:26.567812 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:26.567819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:26.567819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:26.567838 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:26.570153 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:26.570584 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:26.570588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:26.570588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:26.570690 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:26.570899 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:26.570900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:26.570901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:26.571176 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:26.571310 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:26.571445 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 02:37:26.571596 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260811 02:37:26.575088 7016 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:26.575855 7016 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:26.588384 7016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d01139748444826b0ab34b1e13a21f5: Generated new TSK 0
I20260811 02:37:26.588619 7016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:26.626108 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39013
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:26.724792 7032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:26.725045 7032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:26.725083 7032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:26.726485 7032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:26.726541 7032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:26.726584 7032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:26.728432 7032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:26.728754 7032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:26.729007 7032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:26.734598 7039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:26.734592 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:26.734604 7043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:26.734951 7032 server_base.cc:1034] running on GCE node
I20260811 02:37:26.735176 7032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:26.735440 7032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:26.736596 7032 hybrid_clock.cc:648] HybridClock initialized: now 1786415846736568 us; error 37 us; skew 500 ppm
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:26.739800 7032 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:26.740445 7032 webserver.cc:460] Webserver started at http://127.2.251.1:43713/ using document root <none> and password file <none>
I20260811 02:37:26.740713 7032 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:26.740768 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:26.740911 7032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:26.742717 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "ceb6c90d97c049c5983c9039162c1f66"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "3533a514e0726cb489f937cb675adb25"
server_key_iv: "b4b8c70f7b27bf50527bb6497fbd34f9"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.743108 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb6c90d97c049c5983c9039162c1f66"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "3533a514e0726cb489f937cb675adb25"
server_key_iv: "b4b8c70f7b27bf50527bb6497fbd34f9"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.744668 7032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:37:26.745885 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.746203 7032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:26.746311 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "ceb6c90d97c049c5983c9039162c1f66"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "3533a514e0726cb489f937cb675adb25"
server_key_iv: "b4b8c70f7b27bf50527bb6497fbd34f9"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.746387 7032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:26.769418 7032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:26.772024 7032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:26.772204 7032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:26.772473 7032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:26.772826 7032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:26.772862 7032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.772892 7032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:26.772909 7032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.779225 7032 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:44997
I20260811 02:37:26.779291 7250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:44997 every 8 connection(s)
I20260811 02:37:26.779625 7032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:26.784626 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7032
I20260811 02:37:26.784757 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:26.784992 3052 external_mini_cluster.cc:1428] Setting key 1f198f3eca58469ea3d31de14d70f10f
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:26.791050 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:26.780495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:55019 (local address 127.2.251.62:39013)
0811 02:37:26.780599 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:26.780601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:26.780759 (+ 158us) server_negotiation.cc:407] Connection header received
0811 02:37:26.781320 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:26.781321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:26.781340 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:26.781397 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:26.782475 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.783000 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.784029 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.784249 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.784316 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:26.787672 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:26.787674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:26.787675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:26.787685 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:26.789349 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:26.789868 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:26.789870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:26.789870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:26.789954 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:26.790239 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:26.790240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:26.790240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:26.790404 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:26.790471 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:26.790862 (+ 391us) server_negotiation.cc:299] Negotiation successful
0811 02:37:26.790959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 02:37:26.791083 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:26.780376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39013 (local address 127.2.251.1:55019)
0811 02:37:26.780592 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:26.780650 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:26.781190 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:26.781488 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:26.781493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:26.782126 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:26.782353 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:26.782359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.783102 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.783103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:26.783930 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:26.783933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.784335 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.784336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:26.784479 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:26.785174 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:26.785187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:26.787538 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:26.789461 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:26.789474 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:26.789477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:26.789795 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:26.790038 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:26.790039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:26.790039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:26.790166 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:26.790492 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:26.790499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:26.790645 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 02:37:26.790745 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 02:37:26.790893 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260811 02:37:26.791762 7252 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39013
I20260811 02:37:26.791870 7252 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:26.792083 7252 heartbeater.cc:511] Master 127.2.251.62:39013 requested a full tablet report, sending...
I20260811 02:37:26.792749 6933 ts_manager.cc:194] Registered new tserver with Master: ceb6c90d97c049c5983c9039162c1f66 (127.2.251.1:44997)
I20260811 02:37:26.793680 6933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:55019
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:26.818055 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39013
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:26.919255 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:26.919533 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:26.919572 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:26.921031 7259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:26.921092 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:26.921121 7259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:26.922807 7259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:26.923062 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:26.923290 7259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:26.930043 7264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:26.930088 7259 server_base.cc:1034] running on GCE node
W20260811 02:37:26.930143 7266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:26.930047 7271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:26.930521 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:26.930795 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:26.931988 7259 hybrid_clock.cc:648] HybridClock initialized: now 1786415846931979 us; error 52 us; skew 500 ppm
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:26.935129 7259 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:26.935640 7259 webserver.cc:460] Webserver started at http://127.2.251.2:39289/ using document root <none> and password file <none>
I20260811 02:37:26.935871 7259 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:26.935925 7259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:26.936062 7259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:26.937216 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "1793766d25964888b774db947112ccc5"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "58f52f40551b877f796220bb50ef02e5"
server_key_iv: "b57ce7e6791b541da7747fcd346ce69c"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.937927 7259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "1793766d25964888b774db947112ccc5"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "58f52f40551b877f796220bb50ef02e5"
server_key_iv: "b57ce7e6791b541da7747fcd346ce69c"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.940027 7259 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:26.941248 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.941538 7259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:26.941658 7259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "1793766d25964888b774db947112ccc5"
format_stamp: "Formatted at 2026-08-11 02:37:26 on dist-test-slave-cvwc"
server_key: "58f52f40551b877f796220bb50ef02e5"
server_key_iv: "b57ce7e6791b541da7747fcd346ce69c"
server_key_version: "encryptionkey@0"
I20260811 02:37:26.941793 7259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:26.956247 7259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:26.958834 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:26.958994 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:26.959242 7259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:26.959581 7259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:26.959616 7259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.959647 7259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:26.959664 7259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:26.966619 7259 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:32973
I20260811 02:37:26.966688 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:32973 every 8 connection(s)
I20260811 02:37:26.967026 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
Aug 11 02:37:26 dist-test-slave-cvwc krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:26.975244 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7259
I20260811 02:37:26.975342 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:26.975545 3052 external_mini_cluster.cc:1428] Setting key 72df056a7f31ad5553480a917ac528cf
I20260811 02:37:26.978797 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:26.967974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39013 (local address 127.2.251.2:42567)
0811 02:37:26.968354 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:26.968399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:26.968970 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:26.969308 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:26.969312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:26.970024 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:26.970245 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:26.970250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.971024 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.971025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:26.971911 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:26.971914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.972468 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.972469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:26.972606 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:26.973154 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:26.973163 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:26.974899 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:26.976866 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:26.976877 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:26.976880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:26.977224 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:26.977529 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:26.977531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:26.977531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:26.977699 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:26.978102 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:26.978109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:26.978299 (+ 190us) client_negotiation.cc:769] Sending connection context
0811 02:37:26.978439 (+ 140us) client_negotiation.cc:240] Negotiation successful
0811 02:37:26.978618 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":203,"threads_started":1}
I20260811 02:37:26.978793 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:26.968035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:42567 (local address 127.2.251.62:39013)
0811 02:37:26.968186 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:26.968189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:26.968537 (+ 348us) server_negotiation.cc:407] Connection header received
0811 02:37:26.969092 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:26.969094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:26.969106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:26.969175 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:26.970384 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.970915 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.972068 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:26.972316 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:26.972420 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:26.975054 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:26.975056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:26.975056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:26.975065 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:26.976751 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:26.977349 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:26.977350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:26.977350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:26.977432 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:26.977852 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:26.977853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:26.977853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:26.978017 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:26.978092 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:26.978577 (+ 485us) server_negotiation.cc:299] Negotiation successful
0811 02:37:26.978698 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260811 02:37:26.979398 7464 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39013
I20260811 02:37:26.979498 7464 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:26.979686 7464 heartbeater.cc:511] Master 127.2.251.62:39013 requested a full tablet report, sending...
I20260811 02:37:26.980078 6933 ts_manager.cc:194] Registered new tserver with Master: 1793766d25964888b774db947112ccc5 (127.2.251.2:32973)
I20260811 02:37:26.980566 6933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:42567
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:27.009946 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39013
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:27.100566 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:27.100812 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:27.100848 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:27.102322 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:27.102383 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:27.102425 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:27.104238 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:27.104511 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:27.104758 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:27.111195 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:27.111227 7473 server_base.cc:1034] running on GCE node
W20260811 02:37:27.111240 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:27.111195 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:27.111687 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:27.111965 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:27.113166 7473 hybrid_clock.cc:648] HybridClock initialized: now 1786415847113109 us; error 75 us; skew 500 ppm
Aug 11 02:37:27 dist-test-slave-cvwc krb5kdc[6866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:27.116089 7473 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:27.116581 7473 webserver.cc:460] Webserver started at http://127.2.251.3:40225/ using document root <none> and password file <none>
I20260811 02:37:27.116832 7473 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:27.116884 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:27.117012 7473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:27.118211 7473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "05270bcfa458404f8b27d94b97c13d1f"
format_stamp: "Formatted at 2026-08-11 02:37:27 on dist-test-slave-cvwc"
server_key: "9e7d498421c8db2b3e98a959a48e4099"
server_key_iv: "dcfe9c5aaf25d57b01bc4ad9a4d6d48f"
server_key_version: "encryptionkey@0"
I20260811 02:37:27.118604 7473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "05270bcfa458404f8b27d94b97c13d1f"
format_stamp: "Formatted at 2026-08-11 02:37:27 on dist-test-slave-cvwc"
server_key: "9e7d498421c8db2b3e98a959a48e4099"
server_key_iv: "dcfe9c5aaf25d57b01bc4ad9a4d6d48f"
server_key_version: "encryptionkey@0"
I20260811 02:37:27.120023 7473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:27.121024 7501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:27.121306 7473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:27.121462 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "05270bcfa458404f8b27d94b97c13d1f"
format_stamp: "Formatted at 2026-08-11 02:37:27 on dist-test-slave-cvwc"
server_key: "9e7d498421c8db2b3e98a959a48e4099"
server_key_iv: "dcfe9c5aaf25d57b01bc4ad9a4d6d48f"
server_key_version: "encryptionkey@0"
I20260811 02:37:27.121567 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:27.131088 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:27.134243 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:27.134430 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:27.134850 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:27.135263 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:27.135304 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:27.135336 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:27.135354 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:27.141772 7473 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:40555
I20260811 02:37:27.141815 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:40555 every 8 connection(s)
I20260811 02:37:27.142175 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:27.146588 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7473
I20260811 02:37:27.146688 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:27.146873 3052 external_mini_cluster.cc:1428] Setting key b45763ae0be2f10114b283738ea46ab3
Aug 11 02:37:27 dist-test-slave-cvwc krb5kdc[6866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:27.152724 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:27.142976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:47951 (local address 127.2.251.62:39013)
0811 02:37:27.143047 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:27.143049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:27.143294 (+ 245us) server_negotiation.cc:407] Connection header received
0811 02:37:27.143827 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:27.143828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:27.143839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:27.143912 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:27.145387 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:27.146138 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:27.147400 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:27.147657 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:27.147726 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:27.149990 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:27.149992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:27.149993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:27.150001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:27.151360 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:27.151777 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:27.151778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:27.151778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:27.151833 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:27.152062 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:27.152063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:27.152063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:27.152191 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:27.152250 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:27.152531 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 02:37:27.152629 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:37:27.152902 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:27.142927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39013 (local address 127.2.251.3:47951)
0811 02:37:27.143110 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:27.143156 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:27.143667 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:27.144205 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:27.144211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:27.144943 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:27.145178 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:27.145185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:27.146302 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:27.146306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:27.147299 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:27.147301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:27.147736 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:27.147737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:27.147871 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:27.148403 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:27.148413 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:27.149886 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:27.151460 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:27.151468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:27.151471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:27.151695 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:27.151893 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:27.151894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:27.151895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:27.152000 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:27.152258 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:27.152265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:27.152403 (+ 138us) client_negotiation.cc:769] Sending connection context
0811 02:37:27.152496 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 02:37:27.152666 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260811 02:37:27.153420 7681 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39013
I20260811 02:37:27.153512 7681 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:27.153697 7681 heartbeater.cc:511] Master 127.2.251.62:39013 requested a full tablet report, sending...
I20260811 02:37:27.154444 6933 ts_manager.cc:194] Registered new tserver with Master: 05270bcfa458404f8b27d94b97c13d1f (127.2.251.3:40555)
I20260811 02:37:27.155041 6933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:47951
I20260811 02:37:27.158934 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 02:37:27.172787 6990 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:27.171765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.2.251.62:39013)
0811 02:37:27.171867 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:27.171870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:27.171885 (+ 15us) server_negotiation.cc:407] Connection header received
0811 02:37:27.171950 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:27.171952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:27.171967 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:27.172021 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:27.172689 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57842: BlockingRecv error: recv got EOF from 127.0.0.1:57842 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 02:37:27.174371 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7032
I20260811 02:37:27.226680 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7259
I20260811 02:37:27.249846 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7473
I20260811 02:37:27.267231 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6885
2026-08-11T02:37:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2263 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:27 dist-test-slave-cvwc krb5kdc[7701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:27 dist-test-slave-cvwc krb5kdc[7701](info): set up 2 sockets
Aug 11 02:37:27 dist-test-slave-cvwc krb5kdc[7701](info): commencing operation
krb5kdc: starting...
W20260811 02:37:28.528808 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:28 dist-test-slave-cvwc krb5kdc[7701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:28.678077 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:37707
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:37707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:28.777240 7718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:28.777637 7718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:28.777945 7718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:28.780246 7718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:28.780319 7718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:28.780334 7718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:28.780347 7718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:28.780359 7718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:28.783191 7718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:37707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:28.783638 7718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:28.783948 7718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:28.790251 7732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:28.790289 7718 server_base.cc:1034] running on GCE node
W20260811 02:37:28.790313 7730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:28.790254 7735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:28.790860 7718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:28.791220 7718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:28.792411 7718 hybrid_clock.cc:648] HybridClock initialized: now 1786415848792397 us; error 40 us; skew 500 ppm
Aug 11 02:37:28 dist-test-slave-cvwc krb5kdc[7701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415848, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:28.799871 7718 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:28.800503 7718 webserver.cc:460] Webserver started at http://127.2.251.62:33661/ using document root <none> and password file <none>
I20260811 02:37:28.800741 7718 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:28.800792 7718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:28.800925 7718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:28.802744 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "70dd858d319d49b29283d9a748e88085"
format_stamp: "Formatted at 2026-08-11 02:37:28 on dist-test-slave-cvwc"
server_key: "809dda498fffa5647dd66c07bc4fd1c7"
server_key_iv: "57e91c00dcb4dee9c0b1bb722eb45de9"
server_key_version: "encryptionkey@0"
I20260811 02:37:28.803215 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "70dd858d319d49b29283d9a748e88085"
format_stamp: "Formatted at 2026-08-11 02:37:28 on dist-test-slave-cvwc"
server_key: "809dda498fffa5647dd66c07bc4fd1c7"
server_key_iv: "57e91c00dcb4dee9c0b1bb722eb45de9"
server_key_version: "encryptionkey@0"
I20260811 02:37:28.805135 7718 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:28.806375 7746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:28.806792 7718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:28.806972 7718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "70dd858d319d49b29283d9a748e88085"
format_stamp: "Formatted at 2026-08-11 02:37:28 on dist-test-slave-cvwc"
server_key: "809dda498fffa5647dd66c07bc4fd1c7"
server_key_iv: "57e91c00dcb4dee9c0b1bb722eb45de9"
server_key_version: "encryptionkey@0"
I20260811 02:37:28.807116 7718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:28.839852 7718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:28.842777 7718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:28.842943 7718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:28.848675 7718 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:37707
I20260811 02:37:28.848721 7837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:37707 every 8 connection(s)
I20260811 02:37:28.849107 7718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:28.850476 7838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:28.853094 7838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085: Bootstrap starting.
I20260811 02:37:28.853850 7838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:28.854166 7838 log.cc:824] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:28.855355 7838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085: No bootstrap required, opened a new log
I20260811 02:37:28.856853 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7718
I20260811 02:37:28.856971 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:28.857177 3052 external_mini_cluster.cc:1428] Setting key aab7f063a5d58f4e57fc462d9665fbed
I20260811 02:37:28.857885 7838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd858d319d49b29283d9a748e88085" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37707 } }
I20260811 02:37:28.858103 7838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:28.858141 7838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70dd858d319d49b29283d9a748e88085, State: Initialized, Role: FOLLOWER
I20260811 02:37:28.858273 7838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd858d319d49b29283d9a748e88085" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37707 } }
I20260811 02:37:28.858355 7838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:28.858397 7838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:28.858443 7838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:28.859314 7838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd858d319d49b29283d9a748e88085" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37707 } }
I20260811 02:37:28.859441 7838 leader_election.cc:302] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70dd858d319d49b29283d9a748e88085; no voters:
I20260811 02:37:28.859638 7838 leader_election.cc:288] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:28.859898 7846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:28.860172 7846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [term 1 LEADER]: Becoming Leader. State: Replica: 70dd858d319d49b29283d9a748e88085, State: Running, Role: LEADER
I20260811 02:37:28.860344 7846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd858d319d49b29283d9a748e88085" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37707 } }
I20260811 02:37:28.860813 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70dd858d319d49b29283d9a748e88085. Latest consensus state: current_term: 1 leader_uuid: "70dd858d319d49b29283d9a748e88085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd858d319d49b29283d9a748e88085" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37707 } } }
I20260811 02:37:28.860898 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:28.861202 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70dd858d319d49b29283d9a748e88085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dd858d319d49b29283d9a748e88085" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37707 } } }
I20260811 02:37:28.861370 7838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:28.861383 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:28.862007 7849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:28.862895 7849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:28.865218 7849 catalog_manager.cc:1347] Generated new cluster ID: 13869b36914746b5ae242c44ab99d22a
I20260811 02:37:28.865319 7849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:28 dist-test-slave-cvwc krb5kdc[7701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:28.871888 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:28.857988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.2.251.62:37707)
0811 02:37:28.859885 (+ 1897us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:28.859898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:28.859939 (+ 41us) server_negotiation.cc:407] Connection header received
0811 02:37:28.860092 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:28.860099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:28.860165 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:28.861160 (+ 995us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:28.861299 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:28.863176 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:28.864494 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:28.865028 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:28.865156 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:28.868403 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:28.868409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:28.868410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:28.868428 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:28.870246 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:28.870642 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:28.870645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:28.870646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:28.870749 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:28.870943 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:28.870944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:28.870944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:28.871215 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:28.871399 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:28.871503 (+ 104us) server_negotiation.cc:299] Negotiation successful
0811 02:37:28.871652 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1818,"thread_start_us":532,"threads_started":1}
I20260811 02:37:28.873148 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:37:28.881322 7849 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:28.882156 7849 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:28.892911 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:28.889363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.2.251.62:37707)
0811 02:37:28.889416 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:28.889418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:28.889523 (+ 105us) server_negotiation.cc:407] Connection header received
0811 02:37:28.889591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:28.889592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:28.889604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:28.889653 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:28.890104 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:28.890626 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:28.891404 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:28.891633 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:28.891691 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:28.892399 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:28.892401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:28.892401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:28.892638 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:28.892656 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:28.892663 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:28.892714 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:28.892794 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 02:37:28.892824 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:37:28.892946 7849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70dd858d319d49b29283d9a748e88085: Generated new TSK 0
I20260811 02:37:28.893332 7849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:37:28.901914 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7718
2026-08-11T02:37:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1658 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:28 dist-test-slave-cvwc krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:28 dist-test-slave-cvwc krb5kdc[7889](info): set up 2 sockets
Aug 11 02:37:28 dist-test-slave-cvwc krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260811 02:37:30.173560 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:30.327188 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:36763
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:36763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:30.420961 7907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:30.421221 7907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:30.421255 7907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:30.422681 7907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:30.422751 7907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:30.422778 7907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:30.422797 7907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:30.422813 7907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:30.424568 7907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:36763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:30.424883 7907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:30.425139 7907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:30.431423 7915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:30.431684 7914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:30.431730 7907 server_base.cc:1034] running on GCE node
W20260811 02:37:30.431419 7919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:30.432175 7907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:30.432567 7907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:30.433808 7907 hybrid_clock.cc:648] HybridClock initialized: now 1786415850433761 us; error 61 us; skew 500 ppm
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:30.437006 7907 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:30.438067 7907 webserver.cc:460] Webserver started at http://127.2.251.62:42633/ using document root <none> and password file <none>
I20260811 02:37:30.438431 7907 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:30.438526 7907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:30.438975 7907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:30.440992 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "3be62911ff214837aa4ae649c99b3f2c"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "0fb53ae4a547f44c13c947d27f751a1c"
server_key_iv: "1a84db4a259b2bda54d0146576693d35"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.441475 7907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "3be62911ff214837aa4ae649c99b3f2c"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "0fb53ae4a547f44c13c947d27f751a1c"
server_key_iv: "1a84db4a259b2bda54d0146576693d35"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.443394 7907 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260811 02:37:30.444718 7929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.444975 7907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:30.445098 7907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "3be62911ff214837aa4ae649c99b3f2c"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "0fb53ae4a547f44c13c947d27f751a1c"
server_key_iv: "1a84db4a259b2bda54d0146576693d35"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.445194 7907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:30.476559 7907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:30.479347 7907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:30.479528 7907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:30.485081 7907 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:36763
I20260811 02:37:30.485162 8022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:36763 every 8 connection(s)
I20260811 02:37:30.485610 7907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:30.487175 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:30.490595 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c: Bootstrap starting.
I20260811 02:37:30.491320 8023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:30.491634 8023 log.cc:824] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:30.492847 8023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c: No bootstrap required, opened a new log
I20260811 02:37:30.495409 8023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be62911ff214837aa4ae649c99b3f2c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36763 } }
I20260811 02:37:30.495594 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7907
I20260811 02:37:30.495651 8023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:30.495700 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:30.495690 8023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be62911ff214837aa4ae649c99b3f2c, State: Initialized, Role: FOLLOWER
I20260811 02:37:30.495898 3052 external_mini_cluster.cc:1428] Setting key 259f10ce8f6dde6639e36df8555f3036
I20260811 02:37:30.495867 8023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be62911ff214837aa4ae649c99b3f2c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36763 } }
I20260811 02:37:30.495961 8023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:30.496002 8023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:30.496054 8023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:30.496886 8023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be62911ff214837aa4ae649c99b3f2c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36763 } }
I20260811 02:37:30.497011 8023 leader_election.cc:302] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be62911ff214837aa4ae649c99b3f2c; no voters:
I20260811 02:37:30.497171 8023 leader_election.cc:288] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:30.497213 8029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:30.497413 8029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [term 1 LEADER]: Becoming Leader. State: Replica: 3be62911ff214837aa4ae649c99b3f2c, State: Running, Role: LEADER
I20260811 02:37:30.497470 8023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:30.497521 8029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be62911ff214837aa4ae649c99b3f2c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36763 } }
I20260811 02:37:30.498001 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be62911ff214837aa4ae649c99b3f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be62911ff214837aa4ae649c99b3f2c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36763 } } }
I20260811 02:37:30.498095 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:30.498400 8042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:30.498677 8033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be62911ff214837aa4ae649c99b3f2c. Latest consensus state: current_term: 1 leader_uuid: "3be62911ff214837aa4ae649c99b3f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be62911ff214837aa4ae649c99b3f2c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36763 } } }
I20260811 02:37:30.498757 8033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:30.499228 8042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:30.501124 8042 catalog_manager.cc:1347] Generated new cluster ID: 7bc6168d8d044b92a3ee761619f3fa56
I20260811 02:37:30.501183 8042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:30.510589 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:30.496469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.2.251.62:36763)
0811 02:37:30.497081 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:30.497093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:30.497133 (+ 40us) server_negotiation.cc:407] Connection header received
0811 02:37:30.497271 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:30.497279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:30.497366 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:30.498245 (+ 879us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:30.498353 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.500161 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.501067 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.501668 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.501828 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:30.504800 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:30.504805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:30.504805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:30.504821 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:30.507730 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:30.508607 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:30.508612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:30.508613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:30.508783 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:30.509114 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:30.509116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:30.509117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:30.509774 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:30.509979 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:30.510121 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 02:37:30.510295 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":467,"threads_started":1}
I20260811 02:37:30.512694 8042 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:30.513489 8042 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:30.523134 8042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be62911ff214837aa4ae649c99b3f2c: Generated new TSK 0
I20260811 02:37:30.523382 8042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:30.561280 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:36763
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:30.663503 8064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:30.663758 8064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:30.663796 8064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:30.665150 8064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:30.665207 8064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:30.665248 8064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:30.667191 8064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:30.667752 8064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:30.668076 8064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:30.673700 8072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:30.673765 8064 server_base.cc:1034] running on GCE node
W20260811 02:37:30.673871 8074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:30.673759 8071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:30.674122 8064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:30.674404 8064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:30.675594 8064 hybrid_clock.cc:648] HybridClock initialized: now 1786415850675547 us; error 75 us; skew 500 ppm
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:30.679567 8064 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:30.680109 8064 webserver.cc:460] Webserver started at http://127.2.251.1:33231/ using document root <none> and password file <none>
I20260811 02:37:30.680356 8064 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:30.680418 8064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:30.680552 8064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:30.681674 8064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "9a8fb23921c94ed4bda7fd1a51925687"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "8af8b92934bbd075aab4b4486acb0c3f"
server_key_iv: "5d2edbe1d5b50b338702dbd5245e39b3"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.682085 8064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "9a8fb23921c94ed4bda7fd1a51925687"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "8af8b92934bbd075aab4b4486acb0c3f"
server_key_iv: "5d2edbe1d5b50b338702dbd5245e39b3"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.683830 8064 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260811 02:37:30.684867 8089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.685178 8064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:30.685292 8064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "9a8fb23921c94ed4bda7fd1a51925687"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "8af8b92934bbd075aab4b4486acb0c3f"
server_key_iv: "5d2edbe1d5b50b338702dbd5245e39b3"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.685377 8064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:30.713176 8064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:30.716151 8064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:30.716336 8064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:30.716603 8064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:30.716995 8064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:30.717038 8064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.717070 8064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:30.717087 8064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.723459 8064 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:38209
I20260811 02:37:30.723522 8265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:38209 every 8 connection(s)
I20260811 02:37:30.723855 8064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:30.729050 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8064
I20260811 02:37:30.729169 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:30.729406 3052 external_mini_cluster.cc:1428] Setting key a0d293031e91fa5f809e9e6240e12615
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:30.736450 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:30.725040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:43001 (local address 127.2.251.62:36763)
0811 02:37:30.725100 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:30.725103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:30.725346 (+ 243us) server_negotiation.cc:407] Connection header received
0811 02:37:30.725876 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:30.725877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:30.725892 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:30.725965 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:30.727179 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.727712 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.729547 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.729965 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.730036 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:30.732918 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:30.732921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:30.732921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:30.732931 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:30.734553 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:30.735044 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:30.735045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:30.735045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:30.735113 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:30.735378 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:30.735379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:30.735379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:30.735538 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:30.735619 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:30.736231 (+ 612us) server_negotiation.cc:299] Negotiation successful
0811 02:37:30.736343 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 02:37:30.736765 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:30.724950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:36763 (local address 127.2.251.1:43001)
0811 02:37:30.725142 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:30.725188 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:30.725706 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:30.726182 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:30.726186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:30.726827 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:30.727052 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:30.727057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.727927 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.727929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:30.729341 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:30.729349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.730071 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.730072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:30.730265 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:30.731035 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:30.731044 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:30.732796 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:30.734671 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:30.734679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:30.734682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:30.734962 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:30.735182 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:30.735183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:30.735183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:30.735309 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:30.735708 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:30.735731 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:30.736009 (+ 278us) client_negotiation.cc:769] Sending connection context
0811 02:37:30.736178 (+ 169us) client_negotiation.cc:240] Negotiation successful
0811 02:37:30.736401 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260811 02:37:30.737512 8267 heartbeater.cc:347] Connected to a master server at 127.2.251.62:36763
I20260811 02:37:30.737659 8267 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:30.737955 8267 heartbeater.cc:511] Master 127.2.251.62:36763 requested a full tablet report, sending...
I20260811 02:37:30.738607 7961 ts_manager.cc:194] Registered new tserver with Master: 9a8fb23921c94ed4bda7fd1a51925687 (127.2.251.1:38209)
I20260811 02:37:30.739239 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:43001
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:30.765480 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:36763
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:30.866670 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:30.866912 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:30.866940 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:30.868249 8273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:30.868311 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:30.868340 8273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:30.870128 8273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:30.870462 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:30.870712 8273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:30.876724 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:30.876748 8281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:30.876742 8282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:30.877363 8273 server_base.cc:1034] running on GCE node
I20260811 02:37:30.877619 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:30.877926 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:30.879125 8273 hybrid_clock.cc:648] HybridClock initialized: now 1786415850879091 us; error 54 us; skew 500 ppm
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:30.881994 8273 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:30.882942 8273 webserver.cc:460] Webserver started at http://127.2.251.2:42867/ using document root <none> and password file <none>
I20260811 02:37:30.883191 8273 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:30.883239 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:30.883368 8273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:30.884507 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "dc01c19a93e54521bb96bb426840cdc5"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "dc0ac3710990893c2fc0c0ba514cb429"
server_key_iv: "fb21468d7769b874b1afe7d057b6e952"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.884864 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "dc01c19a93e54521bb96bb426840cdc5"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "dc0ac3710990893c2fc0c0ba514cb429"
server_key_iv: "fb21468d7769b874b1afe7d057b6e952"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.886426 8273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:30.887403 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.887727 8273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:30.887830 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "dc01c19a93e54521bb96bb426840cdc5"
format_stamp: "Formatted at 2026-08-11 02:37:30 on dist-test-slave-cvwc"
server_key: "dc0ac3710990893c2fc0c0ba514cb429"
server_key_iv: "fb21468d7769b874b1afe7d057b6e952"
server_key_version: "encryptionkey@0"
I20260811 02:37:30.887898 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:30.934938 8273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:30.937281 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:30.937574 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:30.937915 8273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:30.938268 8273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:30.938308 8273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.938339 8273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:30.938364 8273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:30.944581 8273 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38903
I20260811 02:37:30.944645 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38903 every 8 connection(s)
I20260811 02:37:30.945019 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:30.949780 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8273
I20260811 02:37:30.949934 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:30.950346 3052 external_mini_cluster.cc:1428] Setting key f620e95b23baa31605eaea907b669e03
Aug 11 02:37:30 dist-test-slave-cvwc krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415850, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:30.957188 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:30.945991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:60741 (local address 127.2.251.62:36763)
0811 02:37:30.946124 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:30.946127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:30.946289 (+ 162us) server_negotiation.cc:407] Connection header received
0811 02:37:30.946866 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:30.946868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:30.946881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:30.946944 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:30.948009 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.948523 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.950017 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.950231 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.950310 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:30.953826 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:30.953828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:30.953828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:30.953837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:30.955242 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:30.955863 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:30.955864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:30.955865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:30.955935 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:30.956261 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:30.956262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:30.956262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:30.956421 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:30.956490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:30.956941 (+ 451us) server_negotiation.cc:299] Negotiation successful
0811 02:37:30.957087 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 02:37:30.957310 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:30.945838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:36763 (local address 127.2.251.2:60741)
0811 02:37:30.946144 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:30.946198 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:30.946761 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:30.947035 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:30.947040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:30.947659 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:30.947918 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:30.947924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.948615 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.948617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:30.949898 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:30.949903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:30.950419 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:30.950422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:30.950704 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:30.951511 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:30.951522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:30.953563 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:30.955389 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:30.955402 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:30.955406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:30.955756 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:30.956006 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:30.956007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:30.956007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:30.956160 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:30.956495 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:30.956503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:30.956688 (+ 185us) client_negotiation.cc:769] Sending connection context
0811 02:37:30.956809 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 02:37:30.957018 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260811 02:37:30.958508 8482 heartbeater.cc:347] Connected to a master server at 127.2.251.62:36763
I20260811 02:37:30.958635 8482 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:30.958855 8482 heartbeater.cc:511] Master 127.2.251.62:36763 requested a full tablet report, sending...
I20260811 02:37:30.959434 7961 ts_manager.cc:194] Registered new tserver with Master: dc01c19a93e54521bb96bb426840cdc5 (127.2.251.2:38903)
I20260811 02:37:30.959968 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:60741
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:30.985219 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:36763
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:31.075295 8490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:31.075525 8490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:31.075551 8490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:31.076740 8490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:31.076803 8490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:31.076833 8490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:31.078636 8490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:36763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:31.078923 8490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:31.079133 8490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:31.084862 8506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:31.084831 8501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:31.084829 8502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:31.085090 8490 server_base.cc:1034] running on GCE node
I20260811 02:37:31.085403 8490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:31.085644 8490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:31.086863 8490 hybrid_clock.cc:648] HybridClock initialized: now 1786415851086824 us; error 74 us; skew 500 ppm
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415851, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:31.090252 8490 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:31.090726 8490 webserver.cc:460] Webserver started at http://127.2.251.3:40293/ using document root <none> and password file <none>
I20260811 02:37:31.091059 8490 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:31.091131 8490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:31.091267 8490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:31.092391 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "1daaddbf7c9b40119c8da74273eac28b"
format_stamp: "Formatted at 2026-08-11 02:37:31 on dist-test-slave-cvwc"
server_key: "08984e000a3d47a908191a6cd298ccf3"
server_key_iv: "222c1c93ced51d6295ca1290642065db"
server_key_version: "encryptionkey@0"
I20260811 02:37:31.092712 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "1daaddbf7c9b40119c8da74273eac28b"
format_stamp: "Formatted at 2026-08-11 02:37:31 on dist-test-slave-cvwc"
server_key: "08984e000a3d47a908191a6cd298ccf3"
server_key_iv: "222c1c93ced51d6295ca1290642065db"
server_key_version: "encryptionkey@0"
I20260811 02:37:31.094323 8490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:31.095252 8515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:31.095621 8490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:31.095736 8490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "1daaddbf7c9b40119c8da74273eac28b"
format_stamp: "Formatted at 2026-08-11 02:37:31 on dist-test-slave-cvwc"
server_key: "08984e000a3d47a908191a6cd298ccf3"
server_key_iv: "222c1c93ced51d6295ca1290642065db"
server_key_version: "encryptionkey@0"
I20260811 02:37:31.095816 8490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:31.109402 8490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:31.112283 8490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:31.112474 8490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:31.112727 8490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:31.113075 8490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:31.113112 8490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:31.113144 8490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:31.113161 8490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:31.120823 8490 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:37369
I20260811 02:37:31.120893 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:37369 every 8 connection(s)
I20260811 02:37:31.121290 8490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:31.121796 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8490
I20260811 02:37:31.121896 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:31.122102 3052 external_mini_cluster.cc:1428] Setting key 22b2642a20176d8322333046f8b2e6d9
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415851, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:31.133541 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:31.122131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:60789 (local address 127.2.251.62:36763)
0811 02:37:31.122376 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:31.122388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:31.122526 (+ 138us) server_negotiation.cc:407] Connection header received
0811 02:37:31.123172 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:31.123173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:31.123185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:31.123229 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:31.124284 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:31.124931 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:31.126545 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:31.126752 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:31.126812 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:31.129876 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:31.129878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:31.129879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:31.129890 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:31.131496 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:31.132050 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:31.132052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:31.132053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:31.132142 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:31.132642 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:31.132643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:31.132644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:31.132824 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:31.132900 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:31.133334 (+ 434us) server_negotiation.cc:299] Negotiation successful
0811 02:37:31.133440 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260811 02:37:31.133787 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:31.122122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:36763 (local address 127.2.251.3:60789)
0811 02:37:31.122377 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:31.122422 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:31.123080 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:31.123319 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:31.123323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:31.123951 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:31.124177 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:31.124183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:31.125083 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:31.125084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:31.126432 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:31.126437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:31.126878 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:31.126880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:31.127011 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:31.127665 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:31.127679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:31.129638 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:31.131630 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:31.131640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:31.131643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:31.131920 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:31.132286 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:31.132288 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:31.132289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:31.132517 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:31.132929 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:31.132938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:31.133119 (+ 181us) client_negotiation.cc:769] Sending connection context
0811 02:37:31.133237 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 02:37:31.133488 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260811 02:37:31.134490 8710 heartbeater.cc:347] Connected to a master server at 127.2.251.62:36763
I20260811 02:37:31.134599 8710 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:31.134799 8710 heartbeater.cc:511] Master 127.2.251.62:36763 requested a full tablet report, sending...
I20260811 02:37:31.135459 7961 ts_manager.cc:194] Registered new tserver with Master: 1daaddbf7c9b40119c8da74273eac28b (127.2.251.3:37369)
I20260811 02:37:31.136065 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:60789
I20260811 02:37:31.145859 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415851, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415851, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:31.174456 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:31.166284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.2.251.62:36763)
0811 02:37:31.166376 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:31.166378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:31.166436 (+ 58us) server_negotiation.cc:407] Connection header received
0811 02:37:31.166535 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:31.166536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:31.166549 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:31.166599 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:31.167211 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:31.167720 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:31.168456 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:31.168646 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:31.168703 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:31.171673 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:31.171675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:31.171675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:31.171684 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:31.173136 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:31.173495 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:31.173496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:31.173496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:31.173546 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:31.173748 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:31.173749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:31.173750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:31.173928 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:31.174024 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:31.174143 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 02:37:31.174227 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 02:37:31.174978 7961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41008
I20260811 02:37:31.175884 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8064
I20260811 02:37:31.213744 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8273
I20260811 02:37:31.225296 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8490
I20260811 02:37:31.236469 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7907
2026-08-11T02:37:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2303 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[8729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[8729](info): set up 2 sockets
Aug 11 02:37:31 dist-test-slave-cvwc krb5kdc[8729](info): commencing operation
krb5kdc: starting...
W20260811 02:37:32.468930 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:32 dist-test-slave-cvwc krb5kdc[8729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:32.614594 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:33205
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:32.708362 8750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:32.708636 8750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:32.708680 8750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:32.710041 8750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:32.710099 8750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:32.710119 8750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:32.710136 8750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:32.710151 8750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:32.711962 8750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:32.712297 8750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:32.712549 8750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:32.719547 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:32.719524 8762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:32.719528 8768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:32.719563 8750 server_base.cc:1034] running on GCE node
I20260811 02:37:32.720187 8750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:32.720547 8750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:32.721767 8750 hybrid_clock.cc:648] HybridClock initialized: now 1786415852721745 us; error 44 us; skew 500 ppm
Aug 11 02:37:32 dist-test-slave-cvwc krb5kdc[8729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415852, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:32.724771 8750 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:32.725379 8750 webserver.cc:460] Webserver started at http://127.2.251.62:36753/ using document root <none> and password file <none>
I20260811 02:37:32.725672 8750 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:32.725764 8750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:32.725926 8750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:32.727101 8750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "7b079e7a73c24316b4c1e2cd83634434"
format_stamp: "Formatted at 2026-08-11 02:37:32 on dist-test-slave-cvwc"
server_key: "224393a392cb7c0e1222b0274a69cdcb"
server_key_iv: "5122be6b5e02eb7823a5a3383cf82ca4"
server_key_version: "encryptionkey@0"
I20260811 02:37:32.727492 8750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "7b079e7a73c24316b4c1e2cd83634434"
format_stamp: "Formatted at 2026-08-11 02:37:32 on dist-test-slave-cvwc"
server_key: "224393a392cb7c0e1222b0274a69cdcb"
server_key_iv: "5122be6b5e02eb7823a5a3383cf82ca4"
server_key_version: "encryptionkey@0"
I20260811 02:37:32.729015 8750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:32.730123 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:32.730465 8750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:32.730579 8750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "7b079e7a73c24316b4c1e2cd83634434"
format_stamp: "Formatted at 2026-08-11 02:37:32 on dist-test-slave-cvwc"
server_key: "224393a392cb7c0e1222b0274a69cdcb"
server_key_iv: "5122be6b5e02eb7823a5a3383cf82ca4"
server_key_version: "encryptionkey@0"
I20260811 02:37:32.730652 8750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:32.742091 8750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:32.745143 8750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:32.745332 8750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:32.750106 8750 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:33205
I20260811 02:37:32.750196 8870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:33205 every 8 connection(s)
I20260811 02:37:32.750773 8750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:32.751993 8871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:32.752234 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8750
I20260811 02:37:32.752368 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:32.752579 3052 external_mini_cluster.cc:1428] Setting key 0869b989b8e1562438089a0d6043e7e1
I20260811 02:37:32.755406 8871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434: Bootstrap starting.
I20260811 02:37:32.756101 8871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:32.756390 8871 log.cc:824] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:32.757411 8871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434: No bootstrap required, opened a new log
I20260811 02:37:32.760113 8871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b079e7a73c24316b4c1e2cd83634434" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33205 } }
I20260811 02:37:32.760303 8871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:32.760327 8871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b079e7a73c24316b4c1e2cd83634434, State: Initialized, Role: FOLLOWER
I20260811 02:37:32.760453 8871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b079e7a73c24316b4c1e2cd83634434" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33205 } }
I20260811 02:37:32.760535 8871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:32.760567 8871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:32.760615 8871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 02:37:32 dist-test-slave-cvwc krb5kdc[8729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:32.761314 8871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b079e7a73c24316b4c1e2cd83634434" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33205 } }
I20260811 02:37:32.761446 8871 leader_election.cc:302] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b079e7a73c24316b4c1e2cd83634434; no voters:
I20260811 02:37:32.761641 8871 leader_election.cc:288] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:32.761889 8878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:32.761988 8871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:32.762418 8878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [term 1 LEADER]: Becoming Leader. State: Replica: 7b079e7a73c24316b4c1e2cd83634434, State: Running, Role: LEADER
I20260811 02:37:32.762603 8878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b079e7a73c24316b4c1e2cd83634434" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33205 } }
I20260811 02:37:32.763046 8879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b079e7a73c24316b4c1e2cd83634434. Latest consensus state: current_term: 1 leader_uuid: "7b079e7a73c24316b4c1e2cd83634434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b079e7a73c24316b4c1e2cd83634434" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33205 } } }
I20260811 02:37:32.763139 8879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:32.763912 8890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:32.764812 8890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:32.763653 8877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b079e7a73c24316b4c1e2cd83634434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b079e7a73c24316b4c1e2cd83634434" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33205 } } }
I20260811 02:37:32.766012 8877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:32.766571 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:32.753236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.2.251.62:33205)
0811 02:37:32.753438 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:32.753447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:32.753477 (+ 30us) server_negotiation.cc:407] Connection header received
0811 02:37:32.754009 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:32.754018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:32.754074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:32.754206 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:32.755205 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:32.756456 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:32.757556 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:32.758142 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:32.758261 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:32.761297 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:32.761304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:32.761304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:32.761322 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:32.763976 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:32.764466 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:32.764469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:32.764470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:32.764592 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:32.764944 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:32.764946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:32.764946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:32.765397 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:32.765547 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:32.766185 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 02:37:32.766356 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260811 02:37:32.767518 8890 catalog_manager.cc:1347] Generated new cluster ID: 4c8ef1a736514fa680f7d22236cba947
I20260811 02:37:32.767632 8890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:32.772833 8890 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:32.773567 8890 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:32.780687 8890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b079e7a73c24316b4c1e2cd83634434: Generated new TSK 0
I20260811 02:37:32.780900 8890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:32.814060 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33205
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:32.907699 8911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:32.907963 8911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:32.907989 8911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:32.909255 8911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:32.909299 8911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:32.909327 8911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:32.911118 8911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:32.911386 8911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:32.911598 8911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:32.917222 8920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:32.917258 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:32.917371 8911 server_base.cc:1034] running on GCE node
W20260811 02:37:32.917237 8918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:32.917877 8911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:32.918160 8911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:32.919387 8911 hybrid_clock.cc:648] HybridClock initialized: now 1786415852919348 us; error 56 us; skew 500 ppm
Aug 11 02:37:32 dist-test-slave-cvwc krb5kdc[8729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415852, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:32.922609 8911 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:32.923312 8911 webserver.cc:460] Webserver started at http://127.2.251.1:35823/ using document root <none> and password file <none>
I20260811 02:37:32.923589 8911 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:32.923647 8911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:32.923813 8911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:32.925460 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "45adfdd04ab24497abefa613ae5ce7c9"
format_stamp: "Formatted at 2026-08-11 02:37:32 on dist-test-slave-cvwc"
server_key: "fe8b27e40d15440345aba4e330814313"
server_key_iv: "6613e92ee255159c9a5db2d8a384ed07"
server_key_version: "encryptionkey@0"
I20260811 02:37:32.925984 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "45adfdd04ab24497abefa613ae5ce7c9"
format_stamp: "Formatted at 2026-08-11 02:37:32 on dist-test-slave-cvwc"
server_key: "fe8b27e40d15440345aba4e330814313"
server_key_iv: "6613e92ee255159c9a5db2d8a384ed07"
server_key_version: "encryptionkey@0"
I20260811 02:37:32.927590 8911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:32.928795 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:32.929154 8911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:32.929278 8911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "45adfdd04ab24497abefa613ae5ce7c9"
format_stamp: "Formatted at 2026-08-11 02:37:32 on dist-test-slave-cvwc"
server_key: "fe8b27e40d15440345aba4e330814313"
server_key_iv: "6613e92ee255159c9a5db2d8a384ed07"
server_key_version: "encryptionkey@0"
I20260811 02:37:32.929368 8911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:32.943365 8911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:32.945685 8911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:32.945879 8911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:32.946146 8911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:32.946485 8911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:32.946522 8911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:32.946553 8911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:32.946570 8911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:32.952695 8911 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:40469
I20260811 02:37:32.952761 9117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:40469 every 8 connection(s)
I20260811 02:37:32.953152 8911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:32.960964 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8911
I20260811 02:37:32.961103 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:32.961336 3052 external_mini_cluster.cc:1428] Setting key d4a10dce273f6e296f818ec91aab6939
Aug 11 02:37:32 dist-test-slave-cvwc krb5kdc[8729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415852, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:32.967083 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:32.954409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:59231 (local address 127.2.251.62:33205)
0811 02:37:32.954493 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:32.954495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:32.954713 (+ 218us) server_negotiation.cc:407] Connection header received
0811 02:37:32.956683 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:32.956686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:32.956709 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:32.956776 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:32.958208 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:32.959054 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:32.960584 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:32.960854 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:32.960924 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:32.964104 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:32.964106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:32.964106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:32.964115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:32.965560 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:32.966051 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:32.966052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:32.966053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:32.966114 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:32.966362 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:32.966363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:32.966363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:32.966488 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:32.966544 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:32.966856 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 02:37:32.966982 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:37:32.967240 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:32.954242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33205 (local address 127.2.251.1:59231)
0811 02:37:32.954521 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:32.954594 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:32.956302 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:32.956978 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:32.956986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:32.957532 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:32.958025 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:32.958042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:32.959206 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:32.959208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:32.960481 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:32.960485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:32.961011 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:32.961012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:32.961181 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:32.962084 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:32.962096 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:32.963986 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:32.965678 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:32.965688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:32.965691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:32.965970 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:32.966176 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:32.966177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:32.966177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:32.966290 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:32.966547 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:32.966554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:32.966712 (+ 158us) client_negotiation.cc:769] Sending connection context
0811 02:37:32.966804 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 02:37:32.966999 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260811 02:37:32.967836 9118 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33205
I20260811 02:37:32.967948 9118 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:32.968148 9118 heartbeater.cc:511] Master 127.2.251.62:33205 requested a full tablet report, sending...
I20260811 02:37:32.969045 8807 ts_manager.cc:194] Registered new tserver with Master: 45adfdd04ab24497abefa613ae5ce7c9 (127.2.251.1:40469)
I20260811 02:37:32.969704 8807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:59231
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:32.993360 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33205
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:33.088469 9128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:33.088997 9128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:33.089076 9128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:33.091063 9128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:33.091130 9128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:33.091173 9128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:33.093242 9128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:33.093557 9128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:33.093839 9128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:33.099638 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:33.099658 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:33.099824 9128 server_base.cc:1034] running on GCE node
W20260811 02:37:33.099663 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:33.100133 9128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:33.100427 9128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:33.101755 9128 hybrid_clock.cc:648] HybridClock initialized: now 1786415853101702 us; error 35 us; skew 500 ppm
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:33.105563 9128 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:33.106339 9128 webserver.cc:460] Webserver started at http://127.2.251.2:46593/ using document root <none> and password file <none>
I20260811 02:37:33.106624 9128 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:33.106685 9128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:33.106848 9128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:33.108274 9128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "a0e61c3c63b3430495e92b13899e7f92"
format_stamp: "Formatted at 2026-08-11 02:37:33 on dist-test-slave-cvwc"
server_key: "a243ebbe1b1469a38aa9a5c77f81bab3"
server_key_iv: "aa4f58d947f21fade2d278cd70d8d93d"
server_key_version: "encryptionkey@0"
I20260811 02:37:33.108708 9128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "a0e61c3c63b3430495e92b13899e7f92"
format_stamp: "Formatted at 2026-08-11 02:37:33 on dist-test-slave-cvwc"
server_key: "a243ebbe1b1469a38aa9a5c77f81bab3"
server_key_iv: "aa4f58d947f21fade2d278cd70d8d93d"
server_key_version: "encryptionkey@0"
I20260811 02:37:33.110618 9128 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 02:37:33.111855 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:33.112213 9128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:33.112323 9128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "a0e61c3c63b3430495e92b13899e7f92"
format_stamp: "Formatted at 2026-08-11 02:37:33 on dist-test-slave-cvwc"
server_key: "a243ebbe1b1469a38aa9a5c77f81bab3"
server_key_iv: "aa4f58d947f21fade2d278cd70d8d93d"
server_key_version: "encryptionkey@0"
I20260811 02:37:33.112519 9128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:33.122737 9128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:33.125110 9128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:33.125263 9128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:33.125525 9128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:33.125952 9128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:33.125993 9128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:33.126024 9128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:33.126039 9128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:33.132202 9128 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38489
I20260811 02:37:33.132259 9330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38489 every 8 connection(s)
I20260811 02:37:33.132622 9128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:33.140579 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 9128
I20260811 02:37:33.140681 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:33.140864 3052 external_mini_cluster.cc:1428] Setting key 8869c194313e4389a0838fed55ab9099
I20260811 02:37:33.144125 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:33.133573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:59155 (local address 127.2.251.62:33205)
0811 02:37:33.133636 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:33.133638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:33.133877 (+ 239us) server_negotiation.cc:407] Connection header received
0811 02:37:33.134976 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:33.135058 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:33.135082 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:33.135157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:33.135953 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.136468 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.137563 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.137810 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.137880 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:33.140891 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:33.140892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:33.140893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:33.140902 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:33.142436 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.142900 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.142902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.142902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.142965 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.143211 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.143212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.143213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.143339 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:33.143406 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:33.143847 (+ 441us) server_negotiation.cc:299] Negotiation successful
0811 02:37:33.144032 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 02:37:33.144205 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:33.133517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33205 (local address 127.2.251.2:59155)
0811 02:37:33.133681 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:33.133794 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:33.134796 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:33.135276 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:33.135281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:33.135577 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:33.135800 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:33.135805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.136580 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.136581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:33.137447 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:33.137451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.137885 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.137886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:33.138005 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:33.138708 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:33.138724 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:33.140768 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:33.142530 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:33.142543 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:33.142546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:33.142822 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:33.143022 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:33.143023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:33.143023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:33.143150 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:33.143402 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:33.143411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:33.143573 (+ 162us) client_negotiation.cc:769] Sending connection context
0811 02:37:33.143699 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 02:37:33.143926 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260811 02:37:33.144822 9331 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33205
I20260811 02:37:33.144968 9331 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:33.145231 9331 heartbeater.cc:511] Master 127.2.251.62:33205 requested a full tablet report, sending...
I20260811 02:37:33.145814 8807 ts_manager.cc:194] Registered new tserver with Master: a0e61c3c63b3430495e92b13899e7f92 (127.2.251.2:38489)
I20260811 02:37:33.146258 8807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:59155
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:33.173836 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33205
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:33.265308 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:33.265547 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:33.265574 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:33.266849 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:33.266912 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:33.266943 9338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:33.268586 9338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:33.268839 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:33.269085 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:33.274701 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:33.274844 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:33.274727 9353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:33.277383 9338 server_base.cc:1034] running on GCE node
I20260811 02:37:33.277642 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:33.277932 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:33.279119 9338 hybrid_clock.cc:648] HybridClock initialized: now 1786415853279071 us; error 66 us; skew 500 ppm
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:33.283159 9338 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:33.283847 9338 webserver.cc:460] Webserver started at http://127.2.251.3:44875/ using document root <none> and password file <none>
I20260811 02:37:33.284090 9338 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:33.284142 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:33.284268 9338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:33.285568 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "0bae4619cf284785918d658ccb50207c"
format_stamp: "Formatted at 2026-08-11 02:37:33 on dist-test-slave-cvwc"
server_key: "0c2751ec6a33d12111566b1bf3a28a5b"
server_key_iv: "4cce84acac6b500f641d507a75492d2e"
server_key_version: "encryptionkey@0"
I20260811 02:37:33.286001 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "0bae4619cf284785918d658ccb50207c"
format_stamp: "Formatted at 2026-08-11 02:37:33 on dist-test-slave-cvwc"
server_key: "0c2751ec6a33d12111566b1bf3a28a5b"
server_key_iv: "4cce84acac6b500f641d507a75492d2e"
server_key_version: "encryptionkey@0"
I20260811 02:37:33.287595 9338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:33.288511 9363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:33.288717 9338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:33.288813 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "0bae4619cf284785918d658ccb50207c"
format_stamp: "Formatted at 2026-08-11 02:37:33 on dist-test-slave-cvwc"
server_key: "0c2751ec6a33d12111566b1bf3a28a5b"
server_key_iv: "4cce84acac6b500f641d507a75492d2e"
server_key_version: "encryptionkey@0"
I20260811 02:37:33.288880 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:33.301282 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:33.304383 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:33.304533 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:33.304891 9338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:33.305287 9338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:33.305327 9338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:33.305351 9338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:33.305361 9338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:33.312400 9338 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:41449
I20260811 02:37:33.312641 9555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:41449 every 8 connection(s)
I20260811 02:37:33.312853 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:33.320782 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 9338
I20260811 02:37:33.320921 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:33.321182 3052 external_mini_cluster.cc:1428] Setting key 260d7bc64019fb0b3b7c4131d988a071
I20260811 02:37:33.324455 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:33.313794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:40183 (local address 127.2.251.62:33205)
0811 02:37:33.313852 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:33.313854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:33.314165 (+ 311us) server_negotiation.cc:407] Connection header received
0811 02:37:33.315238 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:33.315240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:33.315254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:33.315329 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:33.316215 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.316725 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.317795 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.317997 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.318058 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:33.320669 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:33.320673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:33.320673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:33.320688 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:33.322732 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.323281 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.323282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.323283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.323366 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.323660 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.323662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.323662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.323802 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:33.323881 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:33.324283 (+ 402us) server_negotiation.cc:299] Negotiation successful
0811 02:37:33.324378 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:37:33.324622 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:33.313694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33205 (local address 127.2.251.3:40183)
0811 02:37:33.314039 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:33.314078 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:33.315083 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:33.315482 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:33.315506 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:33.315868 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:33.316100 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:33.316106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.316833 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.316834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:33.317689 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:33.317691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.318082 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.318083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:33.318210 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:33.318828 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:33.318838 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:33.320477 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:33.322846 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:33.322868 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:33.322870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:33.323176 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:33.323448 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:33.323449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:33.323450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:33.323572 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:33.323941 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:33.323950 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:33.324110 (+ 160us) client_negotiation.cc:769] Sending connection context
0811 02:37:33.324218 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 02:37:33.324382 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":97,"threads_started":1}
I20260811 02:37:33.325453 9558 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33205
I20260811 02:37:33.325599 9558 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:33.326035 9558 heartbeater.cc:511] Master 127.2.251.62:33205 requested a full tablet report, sending...
I20260811 02:37:33.326824 8807 ts_manager.cc:194] Registered new tserver with Master: 0bae4619cf284785918d658ccb50207c (127.2.251.3:41449)
I20260811 02:37:33.327302 8807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:40183
I20260811 02:37:33.333976 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:33.365425 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:33.355247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.2.251.1:40469)
0811 02:37:33.355436 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:33.355442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:33.355451 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:33.355480 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:33.355482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:33.355496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:33.355565 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:33.356187 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.356924 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.358116 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.358603 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.358686 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:33.361561 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:33.361577 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:33.361577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:33.361587 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:33.363479 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.364126 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.364132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.364133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.364239 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.364552 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.364555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.364555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.364792 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:33.364966 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:33.365201 (+ 235us) server_negotiation.cc:299] Negotiation successful
0811 02:37:33.365304 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[8729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:33.374287 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:33.367017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.2.251.62:33205)
0811 02:37:33.367086 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:33.367088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:33.367096 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:33.367122 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:33.367123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:33.367134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:33.367183 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:33.367684 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.368197 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.368895 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:33.369091 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:33.369147 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:33.371676 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:33.371677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:33.371677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:33.371685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:33.372958 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.373313 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.373314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.373315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.373369 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:33.373542 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:33.373543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:33.373543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:33.373782 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:33.373963 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:33.374097 (+ 134us) server_negotiation.cc:299] Negotiation successful
0811 02:37:33.374188 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 02:37:33.374560 8807 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624
I20260811 02:37:33.374956 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8911
I20260811 02:37:33.403888 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 9128
I20260811 02:37:33.422204 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 9338
I20260811 02:37:33.471558 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8750
2026-08-11T02:37:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2258 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[9576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[9576](info): set up 2 sockets
Aug 11 02:37:33 dist-test-slave-cvwc krb5kdc[9576](info): commencing operation
krb5kdc: starting...
W20260811 02:37:34.739804 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:34 dist-test-slave-cvwc krb5kdc[9576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:34.886257 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42467
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:42467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:34.976923 9598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:34.977171 9598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:34.977200 9598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:34.978572 9598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:34.978621 9598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:34.978634 9598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:34.978646 9598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:34.978657 9598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:34.980346 9598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:42467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:34.980581 9598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:34.980785 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:34.985901 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:34.985904 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:34.985901 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:34.986128 9598 server_base.cc:1034] running on GCE node
I20260811 02:37:34.986418 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:34.986722 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:34.987907 9598 hybrid_clock.cc:648] HybridClock initialized: now 1786415854987882 us; error 37 us; skew 500 ppm
Aug 11 02:37:34 dist-test-slave-cvwc krb5kdc[9576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415854, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:34.995102 9598 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:34.995532 9598 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:34.995589 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:34.995707 9598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:34.996899 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "257918aa72b249ad957162afc95d325d"
format_stamp: "Formatted at 2026-08-11 02:37:34 on dist-test-slave-cvwc"
server_key: "ab26bc9568f2e0e63fcbea206bb2d526"
server_key_iv: "10a4ea146ecb41265cb2eab865986b4d"
server_key_version: "encryptionkey@0"
I20260811 02:37:34.997663 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "257918aa72b249ad957162afc95d325d"
format_stamp: "Formatted at 2026-08-11 02:37:34 on dist-test-slave-cvwc"
server_key: "ab26bc9568f2e0e63fcbea206bb2d526"
server_key_iv: "10a4ea146ecb41265cb2eab865986b4d"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.000600 9598 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260811 02:37:35.001822 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.002040 9598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:35.002138 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "257918aa72b249ad957162afc95d325d"
format_stamp: "Formatted at 2026-08-11 02:37:34 on dist-test-slave-cvwc"
server_key: "ab26bc9568f2e0e63fcbea206bb2d526"
server_key_iv: "10a4ea146ecb41265cb2eab865986b4d"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.002207 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:35.015249 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:35.018325 9598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:35.018492 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:35.022928 9598 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:42467
I20260811 02:37:35.022956 9702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:42467 every 8 connection(s)
I20260811 02:37:35.023272 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:35.024415 9703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:35.026921 9703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d: Bootstrap starting.
I20260811 02:37:35.027578 9703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:35.027877 9703 log.cc:824] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:35.028617 9703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d: No bootstrap required, opened a new log
I20260811 02:37:35.030524 9703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257918aa72b249ad957162afc95d325d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42467 } }
I20260811 02:37:35.030694 9703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:35.030731 9703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257918aa72b249ad957162afc95d325d, State: Initialized, Role: FOLLOWER
I20260811 02:37:35.030887 9703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257918aa72b249ad957162afc95d325d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42467 } }
I20260811 02:37:35.030975 9703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:35.031009 9703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:35.031061 9703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:35.031976 9703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257918aa72b249ad957162afc95d325d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42467 } }
I20260811 02:37:35.032083 9703 leader_election.cc:302] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257918aa72b249ad957162afc95d325d; no voters:
I20260811 02:37:35.032286 9703 leader_election.cc:288] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:35.032419 9710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:35.032615 9703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:35.032619 9710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [term 1 LEADER]: Becoming Leader. State: Replica: 257918aa72b249ad957162afc95d325d, State: Running, Role: LEADER
I20260811 02:37:35.032752 9710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257918aa72b249ad957162afc95d325d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42467 } }
I20260811 02:37:35.032982 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 9598
I20260811 02:37:35.033102 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:35.033167 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257918aa72b249ad957162afc95d325d. Latest consensus state: current_term: 1 leader_uuid: "257918aa72b249ad957162afc95d325d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257918aa72b249ad957162afc95d325d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42467 } } }
I20260811 02:37:35.033232 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:35.033299 3052 external_mini_cluster.cc:1428] Setting key 810c96bf42d8cacc15e1c00a4198ff0c
I20260811 02:37:35.033416 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257918aa72b249ad957162afc95d325d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257918aa72b249ad957162afc95d325d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42467 } } }
I20260811 02:37:35.033465 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:35.033576 9727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:35.034440 9727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:35.037288 9727 catalog_manager.cc:1347] Generated new cluster ID: f1512b23f43a4448a8c5a1b21b668605
I20260811 02:37:35.037354 9727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:35.045451 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.033892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.2.251.62:42467)
0811 02:37:35.034251 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.034261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.034301 (+ 40us) server_negotiation.cc:407] Connection header received
0811 02:37:35.034447 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.034456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.034518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.034667 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:35.035499 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.036741 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.037872 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.038556 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.038666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.041588 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:35.041594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:35.041595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:35.041609 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:35.043666 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.044092 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.044096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.044097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.044220 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.044412 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.044413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.044414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.044759 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:35.044930 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.045034 (+ 104us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.045193 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":67,"threads_started":1}
I20260811 02:37:35.058367 9727 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:35.059041 9727 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:35.062091 9727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d: Generated new TSK 0
I20260811 02:37:35.062276 9727 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:35.096675 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42467
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:35.190173 9745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:35.190423 9745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:35.190454 9745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:35.192230 9745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:35.192340 9745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:35.192384 9745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:35.195302 9745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:35.195621 9745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:35.195849 9745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:35.201572 9759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:35.201534 9757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:35.201762 9761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:35.201661 9745 server_base.cc:1034] running on GCE node
I20260811 02:37:35.202059 9745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:35.202373 9745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:35.203557 9745 hybrid_clock.cc:648] HybridClock initialized: now 1786415855203530 us; error 46 us; skew 500 ppm
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:35.207026 9745 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:35.207464 9745 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:35.207521 9745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:35.207626 9745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:35.208730 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "60e683b94cc04f0ba90ac604eb87f647"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "963543700b40963f1f86f922e9c6b660"
server_key_iv: "7fe1acc81aef811145ef6149b57f9cfd"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.209060 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "60e683b94cc04f0ba90ac604eb87f647"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "963543700b40963f1f86f922e9c6b660"
server_key_iv: "7fe1acc81aef811145ef6149b57f9cfd"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.210515 9745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260811 02:37:35.211705 9772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.211957 9745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:35.212064 9745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "60e683b94cc04f0ba90ac604eb87f647"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "963543700b40963f1f86f922e9c6b660"
server_key_iv: "7fe1acc81aef811145ef6149b57f9cfd"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.212133 9745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:35.232131 9745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:35.234725 9745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:35.234859 9745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:35.235100 9745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:35.235430 9745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:35.235471 9745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.235493 9745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:35.235505 9745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.242617 9745 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:34991
I20260811 02:37:35.242681 9964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:34991 every 8 connection(s)
I20260811 02:37:35.242972 9745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:35.243717 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 9745
I20260811 02:37:35.243816 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:35.243997 3052 external_mini_cluster.cc:1428] Setting key bc1f695a216abc1535acd308c3ec9c4a
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:37:35.257419 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.244763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:55697 (local address 127.2.251.62:42467)
0811 02:37:35.244818 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.244821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.244834 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:35.245811 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.245813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.245829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.245888 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:35.247486 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.248333 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.249587 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.249948 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.250033 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.252846 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:35.252847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:35.252847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:35.252901 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:35.254881 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.255773 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.255775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.255775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.255854 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.256494 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.256495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.256496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.256641 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:35.256705 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.257172 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.257269 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:37:35.257468 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.244306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42467 (local address 127.2.251.1:55697)
0811 02:37:35.244586 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.244652 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.245500 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.246002 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.246008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.246444 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:35.247326 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.247336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.248466 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.248467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.249461 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.249466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.250094 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.250096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.250262 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.250968 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:35.250978 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:35.252681 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:35.255136 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:35.255151 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:35.255154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:35.255651 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:35.255984 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:35.255986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:35.255986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:35.256358 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:35.256807 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:35.256816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:35.256981 (+ 165us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.257082 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.257255 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":166,"threads_started":1}
I20260811 02:37:35.258368 9965 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42467
I20260811 02:37:35.258579 9965 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:35.258944 9965 heartbeater.cc:511] Master 127.2.251.62:42467 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
I20260811 02:37:35.259634 9649 ts_manager.cc:194] Registered new tserver with Master: 60e683b94cc04f0ba90ac604eb87f647 (127.2.251.1:34991)
I20260811 02:37:35.260489 9649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:55697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:35.277227 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42467
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:35.376390 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:35.376621 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:35.376650 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:35.377961 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:35.378005 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:35.378034 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:35.379657 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:35.379912 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:35.380127 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:35.386018 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:35.386018 9980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:35.386230 9972 server_base.cc:1034] running on GCE node
W20260811 02:37:35.386287 9979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:35.386684 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:35.387158 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:35.388391 9972 hybrid_clock.cc:648] HybridClock initialized: now 1786415855388379 us; error 44 us; skew 500 ppm
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:35.391177 9972 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:35.391568 9972 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:35.391621 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:35.391736 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:35.392860 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "820dd3973da645b3acc987566dfa6db6"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "54d8bd67bfe8db833e74c0d57e59f286"
server_key_iv: "06a3dd496789d3e41a9af6fb47f110d4"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.393215 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "820dd3973da645b3acc987566dfa6db6"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "54d8bd67bfe8db833e74c0d57e59f286"
server_key_iv: "06a3dd496789d3e41a9af6fb47f110d4"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.394680 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:35.395589 9991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.395828 9972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:35.395941 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "820dd3973da645b3acc987566dfa6db6"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "54d8bd67bfe8db833e74c0d57e59f286"
server_key_iv: "06a3dd496789d3e41a9af6fb47f110d4"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.396013 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:35.420542 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:35.423534 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:35.423774 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:35.424108 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:35.424558 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:35.424611 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.424664 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:35.424700 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.432052 9972 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:33951
I20260811 02:37:35.432247 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:33951 every 8 connection(s)
I20260811 02:37:35.432489 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:35.437248 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 9972
I20260811 02:37:35.437368 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:35.437569 3052 external_mini_cluster.cc:1428] Setting key 7ef2974d95c2f1a9145eeaff5473d8ac
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:35.444262 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.433589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:38719 (local address 127.2.251.62:42467)
0811 02:37:35.433676 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.433678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.434029 (+ 351us) server_negotiation.cc:407] Connection header received
0811 02:37:35.434777 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.434778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.434795 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.434853 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:35.436376 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.436915 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.438192 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.438389 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.438474 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.440982 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:35.440984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:35.440984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:35.440994 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:35.442574 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.443064 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.443065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.443066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.443135 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.443476 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.443477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.443477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.443640 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:35.443717 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.444053 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.444155 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 02:37:35.444478 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.433540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42467 (local address 127.2.251.2:38719)
0811 02:37:35.433819 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.433884 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.434645 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.434982 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.434987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.435359 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:35.436154 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.436172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.437043 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.437044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.438091 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.438095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.438492 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.438493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.438628 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.439179 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:35.439189 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:35.440868 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:35.442688 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:35.442696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:35.442699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:35.442967 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:35.443227 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:35.443228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:35.443229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:35.443391 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:35.443736 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:35.443744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:35.443894 (+ 150us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.444002 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.444233 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":115,"threads_started":1}
I20260811 02:37:35.445262 10185 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42467
I20260811 02:37:35.445374 10185 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:35.445533 10185 heartbeater.cc:511] Master 127.2.251.62:42467 requested a full tablet report, sending...
I20260811 02:37:35.445994 9649 ts_manager.cc:194] Registered new tserver with Master: 820dd3973da645b3acc987566dfa6db6 (127.2.251.2:33951)
I20260811 02:37:35.446373 9649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:38719
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:35.468690 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42467
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:35.570602 10194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:35.570914 10194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:35.571509 10194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:35.573163 10194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:35.573223 10194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:35.573259 10194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:35.575320 10194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:35.575606 10194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:35.575867 10194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:35.581574 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:35.581755 10194 server_base.cc:1034] running on GCE node
W20260811 02:37:35.581600 10204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:35.581574 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:35.582113 10194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:35.582376 10194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:35.583544 10194 hybrid_clock.cc:648] HybridClock initialized: now 1786415855583513 us; error 45 us; skew 500 ppm
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:35.586440 10194 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:35.586788 10194 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:35.586840 10194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:35.586972 10194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:35.588119 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "14090c8b85a34703b9d8ba88544374ce"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "c4b510e2516db7f73690246e1512bbde"
server_key_iv: "47f9e70569ad00d7dc8e7352f74f46ee"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.588485 10194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "14090c8b85a34703b9d8ba88544374ce"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "c4b510e2516db7f73690246e1512bbde"
server_key_iv: "47f9e70569ad00d7dc8e7352f74f46ee"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.589895 10194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:37:35.590811 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.591130 10194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:35.591255 10194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "14090c8b85a34703b9d8ba88544374ce"
format_stamp: "Formatted at 2026-08-11 02:37:35 on dist-test-slave-cvwc"
server_key: "c4b510e2516db7f73690246e1512bbde"
server_key_iv: "47f9e70569ad00d7dc8e7352f74f46ee"
server_key_version: "encryptionkey@0"
I20260811 02:37:35.591441 10194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:35.614290 10194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:35.616879 10194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:35.617065 10194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:35.617349 10194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:35.617692 10194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:35.617779 10194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.617816 10194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:35.617841 10194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:35.624699 10194 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:44613
I20260811 02:37:35.624775 10411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:44613 every 8 connection(s)
I20260811 02:37:35.625180 10194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:35.626843 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 10194
I20260811 02:37:35.626972 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:35.627152 3052 external_mini_cluster.cc:1428] Setting key ee9f3ac87b479ddd1cba0e443f3891f4
Aug 11 02:37:35 dist-test-slave-cvwc krb5kdc[9576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415855, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:35.639146 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.627333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:39747 (local address 127.2.251.62:42467)
0811 02:37:35.627420 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.627423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.627436 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:35.627971 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.627972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.627984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.628037 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:35.629104 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.629622 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.631289 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.631505 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.631584 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.634945 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:35.634949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:35.634949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:35.634961 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:35.636717 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.637407 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.637410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.637410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.637507 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.638063 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.638064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.638064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.638269 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:35.638353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.638907 (+ 554us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.639032 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 02:37:35.639345 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.626983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42467 (local address 127.2.251.3:39747)
0811 02:37:35.627180 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.627236 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.627874 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.628132 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.628136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.628440 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:35.628997 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.629003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.629902 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.629905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.631169 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.631175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.631670 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.631672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.631860 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.632619 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:35.632631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:35.634767 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:35.636874 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:35.636889 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:35.636893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:35.637266 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:35.637655 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:35.637657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:35.637658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:35.637959 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:35.638364 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:35.638376 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:35.638657 (+ 281us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.638778 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.639040 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260811 02:37:35.639968 10412 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42467
I20260811 02:37:35.640133 10412 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:35.640332 10412 heartbeater.cc:511] Master 127.2.251.62:42467 requested a full tablet report, sending...
I20260811 02:37:35.640914 9649 ts_manager.cc:194] Registered new tserver with Master: 14090c8b85a34703b9d8ba88544374ce (127.2.251.3:44613)
I20260811 02:37:35.641422 9649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:39747
I20260811 02:37:35.651439 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:37:35.661376 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.653073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.2.251.62:42467)
0811 02:37:35.653170 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.653172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.653305 (+ 133us) server_negotiation.cc:407] Connection header received
0811 02:37:35.653362 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.653363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.653382 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.653482 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:35.654236 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.654767 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.655626 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.655835 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.655898 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.657685 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:35.657688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:35.657688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:35.657700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:35.659476 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.659990 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.659992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.659992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.660079 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:35.660552 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:35.660554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:35.660554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:35.660811 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:35.660926 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.661154 (+ 228us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.661264 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 02:37:35.664153 9649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:35.666409 9649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:35.674763 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.670140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:44613 (local address 127.0.0.1:43036)
0811 02:37:35.670386 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.670432 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.670512 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.671033 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.671037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.671040 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:35.671133 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.671135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.672258 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.672259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.673409 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.673412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.674390 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.674506 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.674523 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.674591 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.674661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260811 02:37:35.674774 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.670324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.2.251.3:44613)
0811 02:37:35.670816 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.670821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.670832 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:35.670881 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.670884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.670931 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.671114 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:35.671213 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.672149 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.673521 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.674283 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674364 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.674438 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.674625 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.674666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":120,"threads_started":1}
I20260811 02:37:35.675230 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.670426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:33951 (local address 127.0.0.1:57906)
0811 02:37:35.671068 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.671077 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.671120 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.671403 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.671405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.671408 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:35.671472 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.671474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.672835 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.672836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.673837 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.673840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.674924 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.675008 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.675017 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.675065 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.675121 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":10,"thread_start_us":109,"threads_started":1}
I20260811 02:37:35.675307 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.670574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.2.251.1:34991)
0811 02:37:35.670880 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.670883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.670893 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:35.671258 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.671261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.671281 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.671507 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:35.671658 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.672723 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674018 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.674746 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674823 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.674881 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.675172 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.675227 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
I20260811 02:37:35.675647 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.670578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.2.251.2:33951)
0811 02:37:35.670788 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.670793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.671183 (+ 390us) server_negotiation.cc:407] Connection header received
0811 02:37:35.671237 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.671240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.671256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.671340 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:35.671567 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.672690 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.673933 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.674746 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674823 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.674881 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.675503 (+ 622us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.675559 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260811 02:37:35.675235 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.670150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34991 (local address 127.0.0.1:47998)
0811 02:37:35.670775 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.670788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.670948 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.671540 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.671541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.671544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:35.671591 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.671593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.672867 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.672868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.673900 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.673903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.674929 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.674930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.675008 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.675015 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.675065 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.675121 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":12,"thread_start_us":160,"threads_started":1}
I20260811 02:37:35.676157 10298 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8a2ecd870e441698e9fafd89532d85 (DEFAULT_TABLE table=test-table [id=a56742a5d26645f69b8b045971bc0c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:35.676366 10079 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8a2ecd870e441698e9fafd89532d85 (DEFAULT_TABLE table=test-table [id=a56742a5d26645f69b8b045971bc0c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:35.676578 10298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8a2ecd870e441698e9fafd89532d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:35.676733 10079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8a2ecd870e441698e9fafd89532d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:35.680188 10444 tablet_bootstrap.cc:492] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Bootstrap starting.
I20260811 02:37:35.680801 10445 tablet_bootstrap.cc:492] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Bootstrap starting.
I20260811 02:37:35.681414 10444 tablet_bootstrap.cc:654] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:35.681690 10445 tablet_bootstrap.cc:654] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:35.681962 9862 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8a2ecd870e441698e9fafd89532d85 (DEFAULT_TABLE table=test-table [id=a56742a5d26645f69b8b045971bc0c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:35.682286 10444 log.cc:824] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:35.682407 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8a2ecd870e441698e9fafd89532d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:35.682560 10445 log.cc:824] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:35.683274 10444 tablet_bootstrap.cc:492] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: No bootstrap required, opened a new log
I20260811 02:37:35.683382 10444 ts_tablet_manager.cc:1397] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:35.683483 10445 tablet_bootstrap.cc:492] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: No bootstrap required, opened a new log
I20260811 02:37:35.683550 10445 ts_tablet_manager.cc:1397] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:37:35.685503 10449 tablet_bootstrap.cc:492] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: Bootstrap starting.
I20260811 02:37:35.685592 10444 raft_consensus.cc:348] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.685592 10445 raft_consensus.cc:348] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.685838 10444 raft_consensus.cc:374] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:35.685837 10445 raft_consensus.cc:374] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:35.685880 10444 raft_consensus.cc:729] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820dd3973da645b3acc987566dfa6db6, State: Initialized, Role: FOLLOWER
I20260811 02:37:35.685880 10445 raft_consensus.cc:729] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14090c8b85a34703b9d8ba88544374ce, State: Initialized, Role: FOLLOWER
I20260811 02:37:35.686021 10445 consensus_queue.cc:260] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.686017 10444 consensus_queue.cc:260] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.686223 10449 tablet_bootstrap.cc:654] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:35.686367 10444 ts_tablet_manager.cc:1428] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:35.686367 10445 ts_tablet_manager.cc:1428] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:35.686501 10412 heartbeater.cc:503] Master 127.2.251.62:42467 was elected leader, sending a full tablet report...
I20260811 02:37:35.686826 10185 heartbeater.cc:503] Master 127.2.251.62:42467 was elected leader, sending a full tablet report...
I20260811 02:37:35.687594 10449 log.cc:824] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:35.688784 10449 tablet_bootstrap.cc:492] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: No bootstrap required, opened a new log
I20260811 02:37:35.688889 10449 ts_tablet_manager.cc:1397] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:37:35.690514 10449 raft_consensus.cc:348] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.690681 10449 raft_consensus.cc:374] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:35.690706 10449 raft_consensus.cc:729] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e683b94cc04f0ba90ac604eb87f647, State: Initialized, Role: FOLLOWER
I20260811 02:37:35.690809 10449 consensus_queue.cc:260] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.691082 10449 ts_tablet_manager.cc:1428] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:37:35.691129 9965 heartbeater.cc:503] Master 127.2.251.62:42467 was elected leader, sending a full tablet report...
W20260811 02:37:35.743914 9968 tablet.cc:2369] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:35.830236 10450 raft_consensus.cc:482] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:35.830466 10450 raft_consensus.cc:504] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.831059 10450 leader_election.cc:288] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60e683b94cc04f0ba90ac604eb87f647 (127.2.251.1:34991), 820dd3973da645b3acc987566dfa6db6 (127.2.251.2:33951)
I20260811 02:37:35.836181 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.831354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:33951 (local address 127.2.251.3:33865)
0811 02:37:35.831445 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.831459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.831526 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.831891 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.831894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.831929 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:35.832062 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.832066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.833386 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.833388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.834816 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.834822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.835750 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.835753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.835867 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.835874 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.835951 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.836014 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":74}
I20260811 02:37:35.836189 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.831378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34991 (local address 127.2.251.3:42465)
0811 02:37:35.831646 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.831655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:35.831711 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:35.831914 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:35.831915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:35.831918 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:35.832062 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.832066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.833407 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.833408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.834816 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:35.834822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.835754 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.835755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:35.835867 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.835875 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 02:37:35.835951 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:37:35.836014 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":19,"thread_start_us":167,"threads_started":1}
I20260811 02:37:35.836256 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.831506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:42465 (local address 127.2.251.1:34991)
0811 02:37:35.831636 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.831638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.831724 (+ 86us) server_negotiation.cc:407] Connection header received
0811 02:37:35.831762 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.831763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.831777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.831846 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:35.832249 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.833180 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.835002 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.835597 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.835693 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.835708 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.836053 (+ 345us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.836152 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 02:37:35.836256 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.831506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:33865 (local address 127.2.251.2:33951)
0811 02:37:35.831577 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.831580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.831597 (+ 17us) server_negotiation.cc:407] Connection header received
0811 02:37:35.831645 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.831646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.831659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.831797 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:35.832284 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.833146 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.835002 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.835597 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.835693 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.835708 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.836053 (+ 345us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.836152 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 02:37:35.836845 9891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8a2ecd870e441698e9fafd89532d85" candidate_uuid: "14090c8b85a34703b9d8ba88544374ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e683b94cc04f0ba90ac604eb87f647" is_pre_election: true
I20260811 02:37:35.836946 10114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8a2ecd870e441698e9fafd89532d85" candidate_uuid: "14090c8b85a34703b9d8ba88544374ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820dd3973da645b3acc987566dfa6db6" is_pre_election: true
I20260811 02:37:35.837083 9891 raft_consensus.cc:2454] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14090c8b85a34703b9d8ba88544374ce in term 0.
I20260811 02:37:35.837181 10114 raft_consensus.cc:2454] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14090c8b85a34703b9d8ba88544374ce in term 0.
I20260811 02:37:35.837476 10222 leader_election.cc:302] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14090c8b85a34703b9d8ba88544374ce, 60e683b94cc04f0ba90ac604eb87f647; no voters:
I20260811 02:37:35.837653 10450 raft_consensus.cc:2781] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:35.837765 10450 raft_consensus.cc:482] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:35.837798 10450 raft_consensus.cc:3037] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:35.839362 10450 raft_consensus.cc:504] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.839679 10450 leader_election.cc:288] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [CANDIDATE]: Term 1 election: Requested vote from peers 60e683b94cc04f0ba90ac604eb87f647 (127.2.251.1:34991), 820dd3973da645b3acc987566dfa6db6 (127.2.251.2:33951)
I20260811 02:37:35.840054 10114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8a2ecd870e441698e9fafd89532d85" candidate_uuid: "14090c8b85a34703b9d8ba88544374ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820dd3973da645b3acc987566dfa6db6"
I20260811 02:37:35.840078 9891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8a2ecd870e441698e9fafd89532d85" candidate_uuid: "14090c8b85a34703b9d8ba88544374ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e683b94cc04f0ba90ac604eb87f647"
I20260811 02:37:35.840227 9891 raft_consensus.cc:3037] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:35.840229 10114 raft_consensus.cc:3037] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:35.842187 10114 raft_consensus.cc:2454] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14090c8b85a34703b9d8ba88544374ce in term 1.
I20260811 02:37:35.842187 9891 raft_consensus.cc:2454] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14090c8b85a34703b9d8ba88544374ce in term 1.
I20260811 02:37:35.842587 10224 leader_election.cc:302] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14090c8b85a34703b9d8ba88544374ce, 820dd3973da645b3acc987566dfa6db6; no voters:
I20260811 02:37:35.842818 10450 raft_consensus.cc:2781] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:35.843075 10450 raft_consensus.cc:686] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 1 LEADER]: Becoming Leader. State: Replica: 14090c8b85a34703b9d8ba88544374ce, State: Running, Role: LEADER
I20260811 02:37:35.843194 10450 consensus_queue.cc:237] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } }
I20260811 02:37:35.844501 9649 catalog_manager.cc:5495] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 14090c8b85a34703b9d8ba88544374ce (127.2.251.3). New cstate: current_term: 1 leader_uuid: "14090c8b85a34703b9d8ba88544374ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14090c8b85a34703b9d8ba88544374ce" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 44613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:35.862259 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.859518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.2.251.3:44613)
0811 02:37:35.859594 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:35.859596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:35.859604 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:35.859653 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:35.859654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:35.859677 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:35.859740 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:35.859911 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.860512 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.861354 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:35.861562 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:35.861622 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:35.861854 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:35.861988 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:35.862070 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:35.862109 (+ 39us) server_negotiation.cc:299] Negotiation successful
0811 02:37:35.862135 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 02:37:35.864454 10114 raft_consensus.cc:1261] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 1 FOLLOWER]: Refusing update from remote peer 14090c8b85a34703b9d8ba88544374ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:35.864447 9891 raft_consensus.cc:1261] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 1 FOLLOWER]: Refusing update from remote peer 14090c8b85a34703b9d8ba88544374ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:35.864858 10462 consensus_queue.cc:1035] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e683b94cc04f0ba90ac604eb87f647" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:35.865036 10450 consensus_queue.cc:1035] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "820dd3973da645b3acc987566dfa6db6" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 33951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:35.871340 9646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:35.871469 9646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:35.872782 9646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 257918aa72b249ad957162afc95d325d: Sending DeleteTablet for 3 replicas of tablet 7d8a2ecd870e441698e9fafd89532d85
I20260811 02:37:35.873606 10079 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8a2ecd870e441698e9fafd89532d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:35 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:57906
I20260811 02:37:35.873649 10298 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8a2ecd870e441698e9fafd89532d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:35 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:43036
I20260811 02:37:35.873703 9862 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8a2ecd870e441698e9fafd89532d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:35 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:47998
I20260811 02:37:35.874172 10488 tablet_replica.cc:330] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647: stopping tablet replica
I20260811 02:37:35.874195 10485 tablet_replica.cc:330] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: stopping tablet replica
I20260811 02:37:35.874334 10485 raft_consensus.cc:2229] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:35.874511 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 9745
I20260811 02:37:35.874523 10485 raft_consensus.cc:2258] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:35.874558 10488 raft_consensus.cc:2229] T 7d8a2ecd870e441698e9fafd89532d85 P 60e683b94cc04f0ba90ac604eb87f647 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:35.875144 10485 ts_tablet_manager.cc:1905] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:35.877182 10485 ts_tablet_manager.cc:1918] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:35.877278 10485 log.cc:1196] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/7d8a2ecd870e441698e9fafd89532d85
I20260811 02:37:35.877593 10485 ts_tablet_manager.cc:1939] T 7d8a2ecd870e441698e9fafd89532d85 P 820dd3973da645b3acc987566dfa6db6: Deleting consensus metadata
I20260811 02:37:35.878237 9627 catalog_manager.cc:4841] TS 820dd3973da645b3acc987566dfa6db6 (127.2.251.2:33951): tablet 7d8a2ecd870e441698e9fafd89532d85 (table test-table [id=a56742a5d26645f69b8b045971bc0c60]) successfully deleted
I20260811 02:37:35.882934 10486 tablet_replica.cc:330] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: stopping tablet replica
I20260811 02:37:35.883138 10486 raft_consensus.cc:2229] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:35.883335 10486 raft_consensus.cc:2258] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:35.887601 10486 ts_tablet_manager.cc:1905] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:35.889590 10486 ts_tablet_manager.cc:1918] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:35.889781 10486 log.cc:1196] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/7d8a2ecd870e441698e9fafd89532d85
I20260811 02:37:35.890080 10486 ts_tablet_manager.cc:1939] T 7d8a2ecd870e441698e9fafd89532d85 P 14090c8b85a34703b9d8ba88544374ce: Deleting consensus metadata
I20260811 02:37:35.891552 9627 catalog_manager.cc:4841] TS 14090c8b85a34703b9d8ba88544374ce (127.2.251.3:44613): tablet 7d8a2ecd870e441698e9fafd89532d85 (table test-table [id=a56742a5d26645f69b8b045971bc0c60]) successfully deleted
W20260811 02:37:35.900746 9624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:34991 (error 108)
I20260811 02:37:35.900880 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 9972
I20260811 02:37:35.905813 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:35.901646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34991 (local address 127.0.0.1:48010)
0811 02:37:35.905608 (+ 3962us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:35.905740 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3918}
W20260811 02:37:35.911940 9624 catalog_manager.cc:4584] TS 60e683b94cc04f0ba90ac604eb87f647 (127.2.251.1:34991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d8a2ecd870e441698e9fafd89532d85: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34991: connect: Connection refused (error 111)
I20260811 02:37:35.918807 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 10194
I20260811 02:37:35.953786 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 9598
2026-08-11T02:37:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2466 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T02:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:35Z Disabled control of system clock
I20260811 02:37:36.004791 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42641
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 02:37:36.119279 10496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:36.119540 10496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:36.119565 10496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:36.120852 10496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:36.120913 10496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:36.120927 10496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:36.120941 10496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:36.120955 10496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:36.122534 10496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:42641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:36.122859 10496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:36.123263 10496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:36.129541 10514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.129547 10509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:36.129756 10496 server_base.cc:1034] running on GCE node
W20260811 02:37:36.129541 10508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:36.130192 10496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:36.130546 10496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:36.131729 10496 hybrid_clock.cc:648] HybridClock initialized: now 1786415856131706 us; error 43 us; skew 500 ppm
I20260811 02:37:36.133559 10496 webserver.cc:460] Webserver started at http://127.2.251.62:38529/ using document root <none> and password file <none>
I20260811 02:37:36.133917 10496 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:36.133981 10496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:36.134128 10496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:36.135385 10496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "04b20b54d6144ad7b36e9f5f2745ce3c"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "10adec33552bd338ce788dc51c3a0943"
server_key_iv: "0fce3422817209959f4efad2f620fb16"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.135820 10496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "04b20b54d6144ad7b36e9f5f2745ce3c"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "10adec33552bd338ce788dc51c3a0943"
server_key_iv: "0fce3422817209959f4efad2f620fb16"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.137812 10496 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:37:36.139307 10522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.139681 10496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:36.139856 10496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "04b20b54d6144ad7b36e9f5f2745ce3c"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "10adec33552bd338ce788dc51c3a0943"
server_key_iv: "0fce3422817209959f4efad2f620fb16"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.139997 10496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:36.143893 10496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:36.144193 10496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:36.144333 10496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:36.149386 10496 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:42641
I20260811 02:37:36.149475 10607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:42641 every 8 connection(s)
I20260811 02:37:36.149922 10496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:36.150749 10608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:36.154206 10608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c: Bootstrap starting.
I20260811 02:37:36.154934 10608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:36.155253 10608 log.cc:824] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:36.156324 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 10496
I20260811 02:37:36.156491 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:36.156543 10608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c: No bootstrap required, opened a new log
I20260811 02:37:36.156714 3052 external_mini_cluster.cc:1428] Setting key 3a87c6197f01f912e452a7ef36102369
I20260811 02:37:36.159533 10608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42641 } }
I20260811 02:37:36.159739 10608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:36.159775 10608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b20b54d6144ad7b36e9f5f2745ce3c, State: Initialized, Role: FOLLOWER
I20260811 02:37:36.160125 10608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42641 } }
I20260811 02:37:36.160356 10608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:36.160404 10608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:36.160538 10608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:36.161821 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.157554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.2.251.62:42641)
0811 02:37:36.157998 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.158011 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.158057 (+ 46us) server_negotiation.cc:407] Connection header received
0811 02:37:36.159519 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.159532 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.159593 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.159821 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.160817 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.160822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.160823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.160828 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.160887 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.160906 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.161227 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.161279 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.161373 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":122,"threads_started":1}
I20260811 02:37:36.162562 10608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42641 } }
I20260811 02:37:36.162921 10608 leader_election.cc:302] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04b20b54d6144ad7b36e9f5f2745ce3c; no voters:
I20260811 02:37:36.163198 10608 leader_election.cc:288] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:36.163399 10615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:36.163715 10615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [term 1 LEADER]: Becoming Leader. State: Replica: 04b20b54d6144ad7b36e9f5f2745ce3c, State: Running, Role: LEADER
I20260811 02:37:36.163748 10608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:36.163841 10615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42641 } }
I20260811 02:37:36.164434 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b20b54d6144ad7b36e9f5f2745ce3c. Latest consensus state: current_term: 1 leader_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42641 } } }
I20260811 02:37:36.164619 10616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b20b54d6144ad7b36e9f5f2745ce3c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42641 } } }
I20260811 02:37:36.164692 10616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:36.164682 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:36.165004 10627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:36.166026 10627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:36.168480 10627 catalog_manager.cc:1347] Generated new cluster ID: 2dd66b8993084ecb9001222dd91058aa
I20260811 02:37:36.168556 10627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:36.188457 10627 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:36.188722 10627 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:36.196193 10627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c: Generated new TSK 0
I20260811 02:37:36.196511 10627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:37:36.203855 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42641
--builtin_ntp_servers=127.2.251.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 02:37:36.298136 10648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:36.298373 10648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:36.298399 10648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:36.299667 10648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:36.299718 10648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:36.299746 10648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:36.302944 10648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:36.303366 10648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:36.303803 10648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:36.310477 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.310540 10662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.310555 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:36.311055 10648 server_base.cc:1034] running on GCE node
I20260811 02:37:36.311363 10648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:36.311754 10648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:36.312947 10648 hybrid_clock.cc:648] HybridClock initialized: now 1786415856312923 us; error 46 us; skew 500 ppm
I20260811 02:37:36.314716 10648 webserver.cc:460] Webserver started at http://127.2.251.1:38341/ using document root <none> and password file <none>
I20260811 02:37:36.315026 10648 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:36.315099 10648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:36.315245 10648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:36.316546 10648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "059e2563e1574dbeb2821479e4aec66e"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "7a993a7ae622f4a32ea29a5e7559a5d8"
server_key_iv: "e40d706e78224835f92adf0febe02360"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.316879 10648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "059e2563e1574dbeb2821479e4aec66e"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "7a993a7ae622f4a32ea29a5e7559a5d8"
server_key_iv: "e40d706e78224835f92adf0febe02360"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.318558 10648 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260811 02:37:36.319880 10669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.320235 10648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:36.320363 10648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "059e2563e1574dbeb2821479e4aec66e"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "7a993a7ae622f4a32ea29a5e7559a5d8"
server_key_iv: "e40d706e78224835f92adf0febe02360"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.320546 10648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:36.324411 10648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:36.324738 10648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:36.324908 10648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:36.325145 10648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:36.325500 10648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:36.325539 10648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.325562 10648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:36.325573 10648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.333972 10648 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:43861
I20260811 02:37:36.334174 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:43861 every 8 connection(s)
I20260811 02:37:36.334529 10648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:36.338887 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.335805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:45671 (local address 127.2.251.62:42641)
0811 02:37:36.335892 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.335895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.336346 (+ 451us) server_negotiation.cc:407] Connection header received
0811 02:37:36.336940 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.336942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.336949 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.337019 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.338627 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.338628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.338629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.338634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.338653 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.338659 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.338724 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.338803 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.338817 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 02:37:36.339062 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.335795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42641 (local address 127.2.251.1:45671)
0811 02:37:36.336130 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.336182 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.336758 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.337103 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.337107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.337117 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.338390 (+ 1273us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.338390 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.338429 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.338481 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.338732 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.338735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.338738 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.338803 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.338867 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":166,"threads_started":1}
I20260811 02:37:36.339987 10860 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42641
I20260811 02:37:36.340112 10860 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:36.340302 10860 heartbeater.cc:511] Master 127.2.251.62:42641 requested a full tablet report, sending...
I20260811 02:37:36.341073 10551 ts_manager.cc:194] Registered new tserver with Master: 059e2563e1574dbeb2821479e4aec66e (127.2.251.1:43861)
I20260811 02:37:36.342589 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 10648
I20260811 02:37:36.342703 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:36.342897 3052 external_mini_cluster.cc:1428] Setting key 50b31050cc08de890488b0745f738ff2
I20260811 02:37:36.343961 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42641
--builtin_ntp_servers=127.2.251.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 02:37:36.440029 10866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:36.440281 10866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:36.440308 10866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:36.441972 10866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:36.442041 10866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:36.442085 10866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:36.443818 10866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:36.444118 10866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:36.444456 10866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:36.450783 10881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.450860 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.450878 10879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:36.451076 10866 server_base.cc:1034] running on GCE node
I20260811 02:37:36.451552 10866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:36.452001 10866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:36.453192 10866 hybrid_clock.cc:648] HybridClock initialized: now 1786415856453176 us; error 36 us; skew 500 ppm
I20260811 02:37:36.455224 10866 webserver.cc:460] Webserver started at http://127.2.251.2:38445/ using document root <none> and password file <none>
I20260811 02:37:36.455546 10866 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:36.455606 10866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:36.455758 10866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:36.456969 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "daf86c66775e88b3131edf1673aa8d2d"
server_key_iv: "eaab3ce209aac1e63c1e679711ed7af3"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.457367 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "daf86c66775e88b3131edf1673aa8d2d"
server_key_iv: "eaab3ce209aac1e63c1e679711ed7af3"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.459045 10866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:36.460224 10893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.460505 10866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:36.460630 10866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "daf86c66775e88b3131edf1673aa8d2d"
server_key_iv: "eaab3ce209aac1e63c1e679711ed7af3"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.460711 10866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:36.464711 10866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:36.465020 10866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:36.465147 10866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:36.465375 10866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:36.465744 10866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:36.465790 10866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.465827 10866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:36.465852 10866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.472558 10866 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:46875
I20260811 02:37:36.472669 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:46875 every 8 connection(s)
I20260811 02:37:36.473026 10866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:36.476718 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.474167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:53693 (local address 127.2.251.62:42641)
0811 02:37:36.474254 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.474257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.474471 (+ 214us) server_negotiation.cc:407] Connection header received
0811 02:37:36.475000 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.475002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.475008 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.475065 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.476321 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.476324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.476325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.476331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.476365 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.476372 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.476467 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.476636 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.476655 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 02:37:36.476817 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.474108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42641 (local address 127.2.251.2:53693)
0811 02:37:36.474347 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.474385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.474889 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.475127 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.475130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.475140 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.476085 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.476086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.476119 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.476154 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.476480 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.476485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.476489 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.476565 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.476624 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260811 02:37:36.477487 11092 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42641
I20260811 02:37:36.477644 11092 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:36.477901 11092 heartbeater.cc:511] Master 127.2.251.62:42641 requested a full tablet report, sending...
I20260811 02:37:36.478673 10551 ts_manager.cc:194] Registered new tserver with Master: 69419d2d5a9040eaa1bc1dbdd50bf8c1 (127.2.251.2:46875)
I20260811 02:37:36.480556 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 10866
I20260811 02:37:36.480651 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:36.480854 3052 external_mini_cluster.cc:1428] Setting key f0d2464c5d74a2993934f53c5980a707
I20260811 02:37:36.482038 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42641
--builtin_ntp_servers=127.2.251.20:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 02:37:36.584148 11098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:36.584419 11098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:36.584447 11098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:36.586014 11098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:36.586083 11098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:36.586115 11098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:36.587941 11098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:36.588241 11098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:36.588608 11098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:36.594277 11105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.594203 11106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:36.594202 11109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:36.594424 11098 server_base.cc:1034] running on GCE node
I20260811 02:37:36.594736 11098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:36.595126 11098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:36.596354 11098 hybrid_clock.cc:648] HybridClock initialized: now 1786415856596318 us; error 75 us; skew 500 ppm
I20260811 02:37:36.597987 11098 webserver.cc:460] Webserver started at http://127.2.251.3:36703/ using document root <none> and password file <none>
I20260811 02:37:36.598328 11098 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:36.598397 11098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:36.598549 11098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:36.600417 11098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "add4877967b648d484f92dd52fcd18a3"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "489d83623809c72629c83d0fb9879b57"
server_key_iv: "cd92762ae3a676f09bc080d3a42c19a0"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.600876 11098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "add4877967b648d484f92dd52fcd18a3"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "489d83623809c72629c83d0fb9879b57"
server_key_iv: "cd92762ae3a676f09bc080d3a42c19a0"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.602628 11098 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260811 02:37:36.603611 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.603916 11098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:36.604094 11098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "add4877967b648d484f92dd52fcd18a3"
format_stamp: "Formatted at 2026-08-11 02:37:36 on dist-test-slave-cvwc"
server_key: "489d83623809c72629c83d0fb9879b57"
server_key_iv: "cd92762ae3a676f09bc080d3a42c19a0"
server_key_version: "encryptionkey@0"
I20260811 02:37:36.604207 11098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:36.608796 11098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:36.609122 11098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:36.609257 11098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:36.609481 11098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:36.609848 11098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:36.609885 11098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.609923 11098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:36.609948 11098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:36.616894 11098 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:42305
I20260811 02:37:36.617169 11311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:42305 every 8 connection(s)
I20260811 02:37:36.617437 11098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:36.619575 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 11098
I20260811 02:37:36.619704 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:36.619949 3052 external_mini_cluster.cc:1428] Setting key 62b7a9481223ed0c03e2172593adb17d
I20260811 02:37:36.621910 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.618421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:45511 (local address 127.2.251.62:42641)
0811 02:37:36.618563 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.618567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.619082 (+ 515us) server_negotiation.cc:407] Connection header received
0811 02:37:36.619631 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.619633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.619645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.619718 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.621603 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.621604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.621604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.621609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.621627 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.621632 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.621679 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.621821 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.621838 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260811 02:37:36.622093 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.618341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42641 (local address 127.2.251.3:45511)
0811 02:37:36.618834 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.618888 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.619466 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.619919 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.619927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.619945 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.621406 (+ 1461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.621407 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.621449 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.621492 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.621702 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.621706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.621744 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.621820 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.621888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":90,"threads_started":1}
I20260811 02:37:36.622560 11315 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42641
I20260811 02:37:36.622673 11315 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:36.622845 11315 heartbeater.cc:511] Master 127.2.251.62:42641 requested a full tablet report, sending...
I20260811 02:37:36.623246 10551 ts_manager.cc:194] Registered new tserver with Master: add4877967b648d484f92dd52fcd18a3 (127.2.251.3:42305)
I20260811 02:37:36.633351 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:37:36.636488 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.634993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.2.251.62:42641)
0811 02:37:36.635114 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.635117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.635211 (+ 94us) server_negotiation.cc:407] Connection header received
0811 02:37:36.635299 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.635300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.635308 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.635356 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.636226 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.636228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.636228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.636235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.636256 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.636262 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.636324 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.636378 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.636398 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 02:37:36.638782 10551 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:36.640981 10551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:36.645969 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.644289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:46875 (local address 127.0.0.1:39326)
0811 02:37:36.644580 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.644590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.644640 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.644787 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.644791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.644799 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.645454 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.645454 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.645471 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.645497 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.645765 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.645774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.645777 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.645848 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.645863 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":85,"threads_started":1}
I20260811 02:37:36.646247 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.644064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.2.251.3:42305)
0811 02:37:36.644395 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.644409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.644525 (+ 116us) server_negotiation.cc:407] Connection header received
0811 02:37:36.645130 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.645137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.645147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.645238 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.645825 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.645833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.645833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.645838 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.645871 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.645894 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.645951 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.646136 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.646153 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":99,"threads_started":1}
I20260811 02:37:36.646279 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.644396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.2.251.2:46875)
0811 02:37:36.644572 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.644576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.644650 (+ 74us) server_negotiation.cc:407] Connection header received
0811 02:37:36.644687 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.644689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.644701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.644787 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.645571 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.645581 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.645582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.645586 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.645624 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.645651 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.645739 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.646183 (+ 444us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.646200 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260811 02:37:36.646101 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.643971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:42305 (local address 127.0.0.1:43510)
0811 02:37:36.644444 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.644454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.644528 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.645263 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.645264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.645266 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.645752 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.645752 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.645757 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.645766 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.645975 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.645976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.645976 (+ 0us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.646015 (+ 39us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.646026 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":2,"thread_start_us":131,"threads_started":1}
I20260811 02:37:36.646100 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.643971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:43861 (local address 127.0.0.1:43110)
0811 02:37:36.644267 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.644312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.644673 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.645243 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.645244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.645248 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.645642 (+ 394us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.645642 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.645648 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.645656 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.645921 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.645922 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.645923 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.645971 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.645985 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260811 02:37:36.646711 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.644064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.2.251.1:43861)
0811 02:37:36.644284 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.644289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.644397 (+ 108us) server_negotiation.cc:407] Connection header received
0811 02:37:36.644730 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.644735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.644750 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.644819 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.645786 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.645793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.645793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.645798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.645834 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.645859 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.645917 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.645979 (+ 62us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.645995 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":106,"threads_started":1}
I20260811 02:37:36.646996 10979 tablet_service.cc:1467] Processing CreateTablet for tablet 27253a8b4d764c1d8fe272b34bb9ff06 (DEFAULT_TABLE table=test-table [id=a82c5b756c9a4370baf9d2f4a0ee064f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:36.646996 11203 tablet_service.cc:1467] Processing CreateTablet for tablet 27253a8b4d764c1d8fe272b34bb9ff06 (DEFAULT_TABLE table=test-table [id=a82c5b756c9a4370baf9d2f4a0ee064f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:36.647413 10979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27253a8b4d764c1d8fe272b34bb9ff06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:36.647413 11203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27253a8b4d764c1d8fe272b34bb9ff06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:36.647428 10755 tablet_service.cc:1467] Processing CreateTablet for tablet 27253a8b4d764c1d8fe272b34bb9ff06 (DEFAULT_TABLE table=test-table [id=a82c5b756c9a4370baf9d2f4a0ee064f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:36.647771 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27253a8b4d764c1d8fe272b34bb9ff06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:36.650645 11340 tablet_bootstrap.cc:492] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: Bootstrap starting.
I20260811 02:37:36.651100 11344 tablet_bootstrap.cc:492] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Bootstrap starting.
I20260811 02:37:36.651371 11340 tablet_bootstrap.cc:654] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:36.651678 11340 log.cc:824] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:36.651861 11344 tablet_bootstrap.cc:654] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:36.652196 11344 log.cc:824] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:36.652529 11340 tablet_bootstrap.cc:492] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: No bootstrap required, opened a new log
I20260811 02:37:36.652594 11340 ts_tablet_manager.cc:1397] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:36.652577 11343 tablet_bootstrap.cc:492] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Bootstrap starting.
I20260811 02:37:36.652997 11344 tablet_bootstrap.cc:492] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: No bootstrap required, opened a new log
I20260811 02:37:36.653071 11344 ts_tablet_manager.cc:1397] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:36.653244 11343 tablet_bootstrap.cc:654] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:36.653564 11343 log.cc:824] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:36.654397 11343 tablet_bootstrap.cc:492] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: No bootstrap required, opened a new log
I20260811 02:37:36.654471 11343 ts_tablet_manager.cc:1397] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:36.654852 11340 raft_consensus.cc:348] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.655004 11340 raft_consensus.cc:374] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:36.655035 11340 raft_consensus.cc:729] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059e2563e1574dbeb2821479e4aec66e, State: Initialized, Role: FOLLOWER
I20260811 02:37:36.655182 11340 consensus_queue.cc:260] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.655457 11340 ts_tablet_manager.cc:1428] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:36.655608 10860 heartbeater.cc:503] Master 127.2.251.62:42641 was elected leader, sending a full tablet report...
I20260811 02:37:36.655757 11344 raft_consensus.cc:348] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.655922 11344 raft_consensus.cc:374] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:36.655957 11344 raft_consensus.cc:729] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69419d2d5a9040eaa1bc1dbdd50bf8c1, State: Initialized, Role: FOLLOWER
I20260811 02:37:36.656085 11344 consensus_queue.cc:260] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.656386 11092 heartbeater.cc:503] Master 127.2.251.62:42641 was elected leader, sending a full tablet report...
I20260811 02:37:36.656352 11344 ts_tablet_manager.cc:1428] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:36.657493 11343 raft_consensus.cc:348] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.657625 11343 raft_consensus.cc:374] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:36.657645 11343 raft_consensus.cc:729] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add4877967b648d484f92dd52fcd18a3, State: Initialized, Role: FOLLOWER
I20260811 02:37:36.657780 11343 consensus_queue.cc:260] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.658011 11343 ts_tablet_manager.cc:1428] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260811 02:37:36.658044 11315 heartbeater.cc:503] Master 127.2.251.62:42641 was elected leader, sending a full tablet report...
W20260811 02:37:36.723888 11094 tablet.cc:2369] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:37:36.835497 10864 tablet.cc:2369] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:37:36.868202 11316 tablet.cc:2369] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:36.889045 11357 raft_consensus.cc:482] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:36.889194 11357 raft_consensus.cc:504] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.889528 11357 leader_election.cc:288] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 059e2563e1574dbeb2821479e4aec66e (127.2.251.1:43861), 69419d2d5a9040eaa1bc1dbdd50bf8c1 (127.2.251.2:46875)
I20260811 02:37:36.890960 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.889830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:46875 (local address 127.2.251.3:46763)
0811 02:37:36.889910 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.889924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.889988 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.890124 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.890125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.890128 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.890622 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.890623 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.890631 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.890645 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.890833 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.890834 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.890835 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.890874 (+ 39us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.890888 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 02:37:36.890961 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.889927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:46763 (local address 127.2.251.2:46875)
0811 02:37:36.889975 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.889977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.889988 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:36.890058 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.890059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.890065 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.890111 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.890753 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.890755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.890755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.890760 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.890777 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.890780 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.890829 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.890874 (+ 45us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.890888 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:37:36.891057 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.889909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:38039 (local address 127.2.251.1:43861)
0811 02:37:36.889976 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.889978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.890179 (+ 201us) server_negotiation.cc:407] Connection header received
0811 02:37:36.890225 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.890226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.890232 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.890309 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.890842 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.890843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.890844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.890848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.890861 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.890866 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.890924 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.890978 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.890990 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:37:36.891083 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.889830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:43861 (local address 127.2.251.3:38039)
0811 02:37:36.890111 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.890121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:36.890179 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:36.890323 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:36.890324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:36.890326 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:36.890738 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:37:36.890739 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:36.890746 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:37:36.890759 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:36.890928 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:36.890929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:36.890929 (+ 0us) client_negotiation.cc:769] Sending connection context
0811 02:37:36.890970 (+ 41us) client_negotiation.cc:240] Negotiation successful
0811 02:37:36.890983 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":108,"thread_start_us":109,"threads_started":1}
I20260811 02:37:36.891348 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27253a8b4d764c1d8fe272b34bb9ff06" candidate_uuid: "add4877967b648d484f92dd52fcd18a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" is_pre_election: true
I20260811 02:37:36.891446 10787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27253a8b4d764c1d8fe272b34bb9ff06" candidate_uuid: "add4877967b648d484f92dd52fcd18a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059e2563e1574dbeb2821479e4aec66e" is_pre_election: true
I20260811 02:37:36.891522 10987 raft_consensus.cc:2454] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add4877967b648d484f92dd52fcd18a3 in term 0.
I20260811 02:37:36.891603 10787 raft_consensus.cc:2454] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add4877967b648d484f92dd52fcd18a3 in term 0.
I20260811 02:37:36.891772 11128 leader_election.cc:302] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69419d2d5a9040eaa1bc1dbdd50bf8c1, add4877967b648d484f92dd52fcd18a3; no voters:
I20260811 02:37:36.891953 11357 raft_consensus.cc:2781] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:36.892030 11357 raft_consensus.cc:482] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:36.892055 11357 raft_consensus.cc:3037] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:36.892777 11357 raft_consensus.cc:504] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.892926 11357 leader_election.cc:288] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [CANDIDATE]: Term 1 election: Requested vote from peers 059e2563e1574dbeb2821479e4aec66e (127.2.251.1:43861), 69419d2d5a9040eaa1bc1dbdd50bf8c1 (127.2.251.2:46875)
I20260811 02:37:36.893117 10787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27253a8b4d764c1d8fe272b34bb9ff06" candidate_uuid: "add4877967b648d484f92dd52fcd18a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059e2563e1574dbeb2821479e4aec66e"
I20260811 02:37:36.893170 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27253a8b4d764c1d8fe272b34bb9ff06" candidate_uuid: "add4877967b648d484f92dd52fcd18a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1"
I20260811 02:37:36.893215 10787 raft_consensus.cc:3037] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:36.893250 10987 raft_consensus.cc:3037] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:36.894063 10987 raft_consensus.cc:2454] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add4877967b648d484f92dd52fcd18a3 in term 1.
I20260811 02:37:36.894150 10787 raft_consensus.cc:2454] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add4877967b648d484f92dd52fcd18a3 in term 1.
I20260811 02:37:36.894270 11128 leader_election.cc:302] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69419d2d5a9040eaa1bc1dbdd50bf8c1, add4877967b648d484f92dd52fcd18a3; no voters:
I20260811 02:37:36.894368 11357 raft_consensus.cc:2781] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:36.894529 11357 raft_consensus.cc:686] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 1 LEADER]: Becoming Leader. State: Replica: add4877967b648d484f92dd52fcd18a3, State: Running, Role: LEADER
I20260811 02:37:36.894637 11357 consensus_queue.cc:237] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } }
I20260811 02:37:36.895893 10549 catalog_manager.cc:5495] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to add4877967b648d484f92dd52fcd18a3 (127.2.251.3). New cstate: current_term: 1 leader_uuid: "add4877967b648d484f92dd52fcd18a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add4877967b648d484f92dd52fcd18a3" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:36.935019 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.933753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.2.251.3:42305)
0811 02:37:36.933797 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:36.933799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:36.933808 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:36.933903 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:36.933904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:36.933919 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:36.933965 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:36.934610 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:36.934611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:36.934612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:36.934616 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:36.934636 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:36.934641 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:36.934710 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:36.934938 (+ 228us) server_negotiation.cc:299] Negotiation successful
0811 02:37:36.934953 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:37:36.937274 10987 raft_consensus.cc:1261] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 1 FOLLOWER]: Refusing update from remote peer add4877967b648d484f92dd52fcd18a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:36.937270 10787 raft_consensus.cc:1261] T 27253a8b4d764c1d8fe272b34bb9ff06 P 059e2563e1574dbeb2821479e4aec66e [term 1 FOLLOWER]: Refusing update from remote peer add4877967b648d484f92dd52fcd18a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:36.937659 11359 consensus_queue.cc:1035] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059e2563e1574dbeb2821479e4aec66e" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:36.937850 11357 consensus_queue.cc:1035] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69419d2d5a9040eaa1bc1dbdd50bf8c1" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:36.940637 11364 mvcc.cc:205] Tried to move back new op lower bound from 7317159350012030976 to 7317159349840875520. Current Snapshot: MvccSnapshot[applied={T|T < 7317159350012030976}]
I20260811 02:37:36.941694 11363 mvcc.cc:205] Tried to move back new op lower bound from 7317159350012030976 to 7317159349840875520. Current Snapshot: MvccSnapshot[applied={T|T < 7317159350012030976 or (T in {7317159350012030976})}]
I20260811 02:37:36.943120 10549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:36.943238 10549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:36.944115 10549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04b20b54d6144ad7b36e9f5f2745ce3c: Sending DeleteTablet for 3 replicas of tablet 27253a8b4d764c1d8fe272b34bb9ff06
I20260811 02:37:36.944464 10755 tablet_service.cc:1514] Processing DeleteTablet for tablet 27253a8b4d764c1d8fe272b34bb9ff06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:36 UTC) from {username='slave'} at 127.0.0.1:43110
I20260811 02:37:36.944501 11203 tablet_service.cc:1514] Processing DeleteTablet for tablet 27253a8b4d764c1d8fe272b34bb9ff06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:36 UTC) from {username='slave'} at 127.0.0.1:43510
I20260811 02:37:36.944587 10979 tablet_service.cc:1514] Processing DeleteTablet for tablet 27253a8b4d764c1d8fe272b34bb9ff06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:36 UTC) from {username='slave'} at 127.0.0.1:39326
I20260811 02:37:36.944909 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 10648
I20260811 02:37:36.954648 11377 tablet_replica.cc:330] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: stopping tablet replica
I20260811 02:37:36.954838 11377 raft_consensus.cc:2229] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:36.954972 11377 raft_consensus.cc:2258] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:36.955292 11378 tablet_replica.cc:330] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: stopping tablet replica
I20260811 02:37:36.955458 11378 raft_consensus.cc:2229] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:36.955600 11378 raft_consensus.cc:2258] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:36.956169 11378 ts_tablet_manager.cc:1905] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:36.958094 11378 ts_tablet_manager.cc:1918] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:36.958192 11378 log.cc:1196] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/27253a8b4d764c1d8fe272b34bb9ff06
I20260811 02:37:36.958524 11378 ts_tablet_manager.cc:1939] T 27253a8b4d764c1d8fe272b34bb9ff06 P 69419d2d5a9040eaa1bc1dbdd50bf8c1: Deleting consensus metadata
I20260811 02:37:36.959120 10528 catalog_manager.cc:4841] TS 69419d2d5a9040eaa1bc1dbdd50bf8c1 (127.2.251.2:46875): tablet 27253a8b4d764c1d8fe272b34bb9ff06 (table test-table [id=a82c5b756c9a4370baf9d2f4a0ee064f]) successfully deleted
W20260811 02:37:36.960848 10530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:43861 (error 108)
I20260811 02:37:36.961131 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 10866
I20260811 02:37:36.966890 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:36.964186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:43861 (local address 127.0.0.1:43122)
0811 02:37:36.966735 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:36.966831 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:43861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2514}
W20260811 02:37:36.997936 10530 catalog_manager.cc:4584] TS 059e2563e1574dbeb2821479e4aec66e (127.2.251.1:43861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27253a8b4d764c1d8fe272b34bb9ff06: Network error: Client connection negotiation failed: client connection to 127.2.251.1:43861: connect: Connection refused (error 111)
I20260811 02:37:37.000252 11377 ts_tablet_manager.cc:1905] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:37.001953 11377 ts_tablet_manager.cc:1918] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:37.002056 11377 log.cc:1196] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/27253a8b4d764c1d8fe272b34bb9ff06
I20260811 02:37:37.002351 11377 ts_tablet_manager.cc:1939] T 27253a8b4d764c1d8fe272b34bb9ff06 P add4877967b648d484f92dd52fcd18a3: Deleting consensus metadata
I20260811 02:37:37.002895 10528 catalog_manager.cc:4841] TS add4877967b648d484f92dd52fcd18a3 (127.2.251.3:42305): tablet 27253a8b4d764c1d8fe272b34bb9ff06 (table test-table [id=a82c5b756c9a4370baf9d2f4a0ee064f]) successfully deleted
I20260811 02:37:37.007061 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 11098
I20260811 02:37:37.025501 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 10496
2026-08-11T02:37:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1068 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 02:37:37.057047 3052 mini_oidc.cc:152] Starting JWKS server
I20260811 02:37:37.058439 3052 webserver.cc:460] Webserver started at https://127.0.0.1:43611/ using document root <none> and password file <none>
I20260811 02:37:37.058560 3052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 02:37:37.058750 3052 webserver.cc:460] Webserver started at http://127.0.0.1:42755/ using document root <none> and password file <none>
2026-08-11T02:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:37Z Disabled control of system clock
I20260811 02:37:37.070349 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34037
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43611/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/testchainca.cert with env {}
W20260811 02:37:37.164176 11386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:37.164423 11386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:37.164449 11386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:37.165704 11386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:37.165797 11386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:37.165808 11386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:37.165820 11386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:37.165831 11386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 02:37:37.165843 11386 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 02:37:37.165853 11386 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43611/jwks.json
I20260811 02:37:37.167682 11386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43611/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:37.167948 11386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:37.168140 11386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:37.173583 11395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:37.173538 11394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:37.173637 11386 server_base.cc:1034] running on GCE node
W20260811 02:37:37.173542 11397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:37.174248 11386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:37.174621 11386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:37.175784 11386 hybrid_clock.cc:648] HybridClock initialized: now 1786415857175762 us; error 38 us; skew 500 ppm
I20260811 02:37:37.177479 11386 webserver.cc:460] Webserver started at http://127.2.251.62:37779/ using document root <none> and password file <none>
I20260811 02:37:37.177858 11386 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:37.177937 11386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:37.178082 11386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:37.179286 11386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "dbde39421af849f59106a3b667e14558"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "5ad800cad99c252dd3555d1d1493c9ce"
server_key_iv: "716c899d9b27ad659343e8c58a0edcbc"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.179704 11386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "dbde39421af849f59106a3b667e14558"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "5ad800cad99c252dd3555d1d1493c9ce"
server_key_iv: "716c899d9b27ad659343e8c58a0edcbc"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.181459 11386 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 02:37:37.182847 11410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:37.183248 11386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:37.183384 11386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "dbde39421af849f59106a3b667e14558"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "5ad800cad99c252dd3555d1d1493c9ce"
server_key_iv: "716c899d9b27ad659343e8c58a0edcbc"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.183492 11386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:37.200083 11386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:37.200493 11386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:37.200646 11386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:37.205510 11386 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:34037
I20260811 02:37:37.205862 11500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:34037 every 8 connection(s)
I20260811 02:37:37.205962 11386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:37.207436 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:37.208009 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 11386
I20260811 02:37:37.208112 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:37.208348 3052 external_mini_cluster.cc:1428] Setting key 70f22ae0f3b60f07f97f77373eb9e3e4
I20260811 02:37:37.210625 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558: Bootstrap starting.
I20260811 02:37:37.211256 11501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:37.211540 11501 log.cc:824] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:37.212239 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558: No bootstrap required, opened a new log
I20260811 02:37:37.214092 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:37.209116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.2.251.62:34037)
0811 02:37:37.209361 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:37.209371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:37.209422 (+ 51us) server_negotiation.cc:407] Connection header received
0811 02:37:37.209921 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:37.209929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:37.209996 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:37.210128 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:37.210336 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.211245 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.212079 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.212536 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.212652 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.213458 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:37.213462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:37.213462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:37.213634 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:37.213668 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:37.213679 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:37.213758 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:37.213836 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 02:37:37.213922 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260811 02:37:37.214350 11501 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbde39421af849f59106a3b667e14558" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34037 } }
I20260811 02:37:37.214504 11501 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:37.214536 11501 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbde39421af849f59106a3b667e14558, State: Initialized, Role: FOLLOWER
I20260811 02:37:37.214660 11501 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbde39421af849f59106a3b667e14558" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34037 } }
I20260811 02:37:37.214751 11501 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:37.214792 11501 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:37.214835 11501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:37.215068 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:37:37.215531 11501 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbde39421af849f59106a3b667e14558" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34037 } }
I20260811 02:37:37.215651 11501 leader_election.cc:302] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbde39421af849f59106a3b667e14558; no voters:
I20260811 02:37:37.215831 11501 leader_election.cc:288] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:37.215869 11506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:37.215997 11506 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [term 1 LEADER]: Becoming Leader. State: Replica: dbde39421af849f59106a3b667e14558, State: Running, Role: LEADER
I20260811 02:37:37.216084 11506 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbde39421af849f59106a3b667e14558" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34037 } }
I20260811 02:37:37.216112 11501 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:37.216450 11506 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbde39421af849f59106a3b667e14558. Latest consensus state: current_term: 1 leader_uuid: "dbde39421af849f59106a3b667e14558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbde39421af849f59106a3b667e14558" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34037 } } }
I20260811 02:37:37.216532 11506 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:37.216459 11511 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbde39421af849f59106a3b667e14558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbde39421af849f59106a3b667e14558" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34037 } } }
I20260811 02:37:37.216720 11511 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558 [sys.catalog]: This master's current role is: LEADER
W20260811 02:37:37.217864 11528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:37:37.217912 11528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:37:37.217970 11515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:37.218154 11515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:37.219805 11515 catalog_manager.cc:1347] Generated new cluster ID: 3f288016f0d246f79c6aa5c5fbbfa861
I20260811 02:37:37.219866 11515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:37.255667 11515 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:37.256222 11515 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:37.261651 11515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbde39421af849f59106a3b667e14558: Generated new TSK 0
I20260811 02:37:37.262061 11515 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 02:37:37.291262 11542 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 02:37:37.291420 11542 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 02:37:37.291726 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:37.274417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.2.251.62:34037)
0811 02:37:37.274480 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:37.274482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:37.274590 (+ 108us) server_negotiation.cc:407] Connection header received
0811 02:37:37.274671 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:37.274673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:37.274685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:37.274769 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0811 02:37:37.274991 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.275770 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.276558 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.276842 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.276916 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.277213 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 02:37:37.291443 (+ 14230us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 02:37:37.291646 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41024: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260811 02:37:37.291796 11504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41024: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 02:37:37.292179 11539 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:37.274339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34037 (local address 127.0.0.1:41024)
0811 02:37:37.274516 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:37.274526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:37.274593 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:37.274826 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:37.274827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:37.274836 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 02:37:37.274913 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:37.274915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.275874 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.275875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:37.276473 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:37.276475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.276965 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.276967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:37.277055 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.277079 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 02:37:37.291656 (+ 14577us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 02:37:37.291778 (+ 122us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.251.62:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.251.62:34037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43611/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 02:37:37.292764 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 11386
2026-08-11T02:37:37Z chronyd exiting
I20260811 02:37:37.352032 3052 test_util.cc:175] -----------------------------------------------
I20260811 02:37:37.352128 3052 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786415831315317-3052-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 02:37:37.352877 3052 mini_oidc.cc:152] Starting JWKS server
I20260811 02:37:37.353817 3052 webserver.cc:460] Webserver started at https://127.0.0.1:38391/ using document root <none> and password file <none>
I20260811 02:37:37.353878 3052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 02:37:37.354012 3052 webserver.cc:460] Webserver started at http://127.0.0.1:44415/ using document root <none> and password file <none>
2026-08-11T02:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:37Z Disabled control of system clock
I20260811 02:37:37.364099 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:36219
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:36219
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38391/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/testchainca.cert with env {}
W20260811 02:37:37.462385 11550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:37.462652 11550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:37.462690 11550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:37.463997 11550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:37.464051 11550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:37.464068 11550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:37.464085 11550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:37.464100 11550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 02:37:37.464118 11550 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 02:37:37.464138 11550 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38391/jwks.json
I20260811 02:37:37.465924 11550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:36219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:36219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38391/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:37.466230 11550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:37.466462 11550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:37.472223 11564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:37.472302 11562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:37.472450 11550 server_base.cc:1034] running on GCE node
W20260811 02:37:37.472528 11559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:37.472826 11550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:37.473203 11550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:37.474386 11550 hybrid_clock.cc:648] HybridClock initialized: now 1786415857474357 us; error 61 us; skew 500 ppm
I20260811 02:37:37.475886 11550 webserver.cc:460] Webserver started at http://127.2.251.62:43827/ using document root <none> and password file <none>
I20260811 02:37:37.476130 11550 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:37.476177 11550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:37.476277 11550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:37.477385 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "9c6a52bf22626e0bb3aa921908cb4bf5"
server_key_iv: "b2a4e68b3fd13325c6a45c6111853668"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.477766 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "9c6a52bf22626e0bb3aa921908cb4bf5"
server_key_iv: "b2a4e68b3fd13325c6a45c6111853668"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.479241 11550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:37.480422 11582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:37.480684 11550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:37.480779 11550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "9c6a52bf22626e0bb3aa921908cb4bf5"
server_key_iv: "b2a4e68b3fd13325c6a45c6111853668"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.480866 11550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:37.503480 11550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:37.503947 11550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:37.504112 11550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:37.508997 11550 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:36219
I20260811 02:37:37.509042 11667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:36219 every 8 connection(s)
I20260811 02:37:37.509408 11550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:37.510561 11668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:37.512789 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 11550
I20260811 02:37:37.512930 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:37.513146 3052 external_mini_cluster.cc:1428] Setting key b6407895084844219980b83322e161df
I20260811 02:37:37.513669 11668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf: Bootstrap starting.
I20260811 02:37:37.514434 11668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:37.514741 11668 log.cc:824] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:37.515616 11668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf: No bootstrap required, opened a new log
I20260811 02:37:37.518260 11668 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36219 } }
I20260811 02:37:37.518479 11668 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:37.518512 11668 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbaeaf7b1df14fc995bc5ee8ed0b44bf, State: Initialized, Role: FOLLOWER
I20260811 02:37:37.518646 11668 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36219 } }
I20260811 02:37:37.518728 11668 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:37.518761 11668 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:37.518810 11668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:37.519593 11668 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36219 } }
I20260811 02:37:37.519711 11668 leader_election.cc:302] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbaeaf7b1df14fc995bc5ee8ed0b44bf; no voters:
I20260811 02:37:37.519873 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:37.513802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.2.251.62:36219)
0811 02:37:37.514022 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:37.514030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:37.514057 (+ 27us) server_negotiation.cc:407] Connection header received
0811 02:37:37.514579 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:37.514588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:37.514666 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:37.514792 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:37.515078 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.516096 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.517312 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.517835 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.518119 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.519029 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:37.519033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:37.519033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:37.519218 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:37.519254 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:37.519265 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:37.519321 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:37.519431 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 02:37:37.519610 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260811 02:37:37.519919 11668 leader_election.cc:288] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:37.519951 11674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:37.520059 11674 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [term 1 LEADER]: Becoming Leader. State: Replica: cbaeaf7b1df14fc995bc5ee8ed0b44bf, State: Running, Role: LEADER
I20260811 02:37:37.520141 11674 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36219 } }
I20260811 02:37:37.520207 11668 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:37.520572 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbaeaf7b1df14fc995bc5ee8ed0b44bf. Latest consensus state: current_term: 1 leader_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36219 } } }
I20260811 02:37:37.520651 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:37.520635 11676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbaeaf7b1df14fc995bc5ee8ed0b44bf" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 36219 } } }
I20260811 02:37:37.520807 11676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:37.520970 11684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:37.521556 11684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:37.521917 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:37:37.527655 11684 catalog_manager.cc:1347] Generated new cluster ID: 9307c2616a924d06a95867252125bdd8
I20260811 02:37:37.527763 11684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:37.551476 11684 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:37.552218 11684 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:37.555842 11684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbaeaf7b1df14fc995bc5ee8ed0b44bf: Generated new TSK 0
I20260811 02:37:37.556155 11684 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 02:37:37.587687 11713 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 02:37:37.587792 11713 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 02:37:37.588131 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:37.564634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.2.251.62:36219)
0811 02:37:37.564696 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:37.564698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:37.564897 (+ 199us) server_negotiation.cc:407] Connection header received
0811 02:37:37.564974 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:37.564975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:37.564986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:37.565051 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0811 02:37:37.565186 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.565771 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.566811 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.567051 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.567112 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.567302 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 02:37:37.587826 (+ 20524us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 02:37:37.588051 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":32}
W20260811 02:37:37.588218 11672 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260811 02:37:37.588259 11710 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:37.564527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:36219 (local address 127.0.0.1:55762)
0811 02:37:37.564782 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:37.564794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:37.564888 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:37.565054 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:37.565055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:37.565057 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0811 02:37:37.565124 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:37.565128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.565883 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.565884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:37.566709 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:37.566713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.567124 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.567125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:37.567189 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.567195 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 02:37:37.588027 (+ 20832us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 02:37:37.588162 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.251.62:36219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38391/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":152,"threads_started":1}
I20260811 02:37:37.588577 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 11550
2026-08-11T02:37:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (286 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 02:37:37.639750 3052 mini_oidc.cc:152] Starting JWKS server
I20260811 02:37:37.640628 3052 webserver.cc:460] Webserver started at https://127.0.0.1:46507/ using document root <none> and password file <none>
I20260811 02:37:37.640681 3052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 02:37:37.640842 3052 webserver.cc:460] Webserver started at http://127.0.0.1:34647/ using document root <none> and password file <none>
2026-08-11T02:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:37Z Disabled control of system clock
I20260811 02:37:37.651237 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34155
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34155
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46507/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 02:37:37.742357 11723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:37.742628 11723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:37.742653 11723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:37.743949 11723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:37.744009 11723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:37.744022 11723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:37.744035 11723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:37.744048 11723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 02:37:37.744060 11723 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 02:37:37.744071 11723 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46507/jwks.json
I20260811 02:37:37.745610 11723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46507/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:37.745901 11723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:37.746196 11723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:37.751609 11733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:37.751634 11723 server_base.cc:1034] running on GCE node
W20260811 02:37:37.751636 11731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:37.751609 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:37.752066 11723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:37.752434 11723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:37.753635 11723 hybrid_clock.cc:648] HybridClock initialized: now 1786415857753605 us; error 49 us; skew 500 ppm
I20260811 02:37:37.755263 11723 webserver.cc:460] Webserver started at http://127.2.251.62:43475/ using document root <none> and password file <none>
I20260811 02:37:37.755523 11723 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:37.755581 11723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:37.755717 11723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:37.756973 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "9f791d8295c741b18a7ea0a2bcfdc629"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "c69e681273ff0bab9be237327055d6eb"
server_key_iv: "214871d096ce556e3112117d51111e93"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.757349 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "9f791d8295c741b18a7ea0a2bcfdc629"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "c69e681273ff0bab9be237327055d6eb"
server_key_iv: "214871d096ce556e3112117d51111e93"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.758845 11723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:37.760267 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:37.760680 11723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:37.760840 11723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "9f791d8295c741b18a7ea0a2bcfdc629"
format_stamp: "Formatted at 2026-08-11 02:37:37 on dist-test-slave-cvwc"
server_key: "c69e681273ff0bab9be237327055d6eb"
server_key_iv: "214871d096ce556e3112117d51111e93"
server_key_version: "encryptionkey@0"
I20260811 02:37:37.761013 11723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:37.784929 11723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:37.785306 11723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:37.785458 11723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:37.790954 11723 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:34155
I20260811 02:37:37.791100 11830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:34155 every 8 connection(s)
I20260811 02:37:37.791512 11723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:37.792414 11831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:37.795804 11831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629: Bootstrap starting.
I20260811 02:37:37.796521 11831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:37.796841 11831 log.cc:824] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:37.797827 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 11723
I20260811 02:37:37.797947 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:37.798110 11831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629: No bootstrap required, opened a new log
I20260811 02:37:37.798132 3052 external_mini_cluster.cc:1428] Setting key ecb4423859d52181b1c81d185a7ffcc1
I20260811 02:37:37.800981 11831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34155 } }
I20260811 02:37:37.801246 11831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:37.801458 11831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f791d8295c741b18a7ea0a2bcfdc629, State: Initialized, Role: FOLLOWER
I20260811 02:37:37.801815 11831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34155 } }
I20260811 02:37:37.802067 11831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:37.802183 11831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:37.802256 11831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:37.803822 11831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34155 } }
I20260811 02:37:37.803994 11831 leader_election.cc:302] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f791d8295c741b18a7ea0a2bcfdc629; no voters:
I20260811 02:37:37.804194 11831 leader_election.cc:288] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:37.804262 11841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:37.804507 11841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [term 1 LEADER]: Becoming Leader. State: Replica: 9f791d8295c741b18a7ea0a2bcfdc629, State: Running, Role: LEADER
I20260811 02:37:37.804634 11841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34155 } }
I20260811 02:37:37.805111 11831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:37.805279 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34155 } } }
I20260811 02:37:37.805279 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f791d8295c741b18a7ea0a2bcfdc629. Latest consensus state: current_term: 1 leader_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f791d8295c741b18a7ea0a2bcfdc629" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34155 } } }
I20260811 02:37:37.805459 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:37.805462 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:37.805855 11846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:37.806849 11846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:37.809111 11846 catalog_manager.cc:1347] Generated new cluster ID: 2410dec025ba40fabcdcc95534fafec0
I20260811 02:37:37.809207 11846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:37.810158 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:37.798798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.2.251.62:34155)
0811 02:37:37.799093 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:37.799103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:37.799138 (+ 35us) server_negotiation.cc:407] Connection header received
0811 02:37:37.800914 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:37.800929 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:37.801054 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:37.801414 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:37.801625 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.804271 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.806549 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.807529 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.808252 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.808735 (+ 483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:37.808739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:37.808740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:37.808961 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:37.809018 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:37.809039 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:37.809144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:37.809793 (+ 649us) server_negotiation.cc:299] Negotiation successful
0811 02:37:37.809941 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":143,"threads_started":1}
I20260811 02:37:37.811326 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:37:37.820927 11846 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:37.821864 11846 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:37.830148 11846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f791d8295c741b18a7ea0a2bcfdc629: Generated new TSK 0
I20260811 02:37:37.830453 11846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:37:37.847344 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:37.842074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.2.251.62:34155)
0811 02:37:37.842131 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:37.842134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:37.842462 (+ 328us) server_negotiation.cc:407] Connection header received
0811 02:37:37.842557 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:37.842559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:37.842580 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:37.842697 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0811 02:37:37.842926 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.843613 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.844882 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.845153 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.845244 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.845536 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 02:37:37.847040 (+ 1504us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 02:37:37.847227 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260811 02:37:37.847440 11838 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 02:37:37.849566 11876 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:37.841931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34155 (local address 127.0.0.1:60720)
0811 02:37:37.842329 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:37.842342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:37.842444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:37.842716 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:37.842717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:37.842720 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0811 02:37:37.842797 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:37.842800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.843726 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.843728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:37.844730 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:37.844736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:37.845278 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:37.845280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:37.845355 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:37.845363 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 02:37:37.847194 (+ 1831us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 02:37:37.849431 (+ 2237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.251.62:34155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1898,"thread_start_us":250,"threads_started":1}
I20260811 02:37:37.849861 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 11723
E20260811 02:37:37.878227 11878 webserver.cc:533] Webserver: sslize failed:
E20260811 02:37:37.878332 11878 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-11T02:37:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (269 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T02:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:37Z Disabled control of system clock
I20260811 02:37:37.918886 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40425
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40425
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/testchainca.cert with env {}
W20260811 02:37:38.014581 11885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:38.014839 11885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:38.014864 11885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:38.016218 11885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:38.016278 11885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:38.016290 11885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:38.016301 11885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:38.016314 11885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 02:37:38.016323 11885 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 02:37:38.016333 11885 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 02:37:38.018141 11885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:38.018422 11885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:38.018680 11885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:38.024226 11894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:38.024257 11891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:38.024235 11897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:38.024621 11885 server_base.cc:1034] running on GCE node
I20260811 02:37:38.024921 11885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:38.025282 11885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:38.026479 11885 hybrid_clock.cc:648] HybridClock initialized: now 1786415858026439 us; error 55 us; skew 500 ppm
I20260811 02:37:38.027945 11885 webserver.cc:460] Webserver started at http://127.2.251.62:46015/ using document root <none> and password file <none>
I20260811 02:37:38.028234 11885 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:38.028294 11885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:38.028430 11885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:38.030128 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "f9604711050349408ca7da63fe78ca1c"
format_stamp: "Formatted at 2026-08-11 02:37:38 on dist-test-slave-cvwc"
server_key: "7e255b10631b2b762155f4eae1217980"
server_key_iv: "bcc871f74896003d28d49ced654ddb2a"
server_key_version: "encryptionkey@0"
I20260811 02:37:38.030599 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "f9604711050349408ca7da63fe78ca1c"
format_stamp: "Formatted at 2026-08-11 02:37:38 on dist-test-slave-cvwc"
server_key: "7e255b10631b2b762155f4eae1217980"
server_key_iv: "bcc871f74896003d28d49ced654ddb2a"
server_key_version: "encryptionkey@0"
I20260811 02:37:38.032490 11885 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:38.033761 11908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:38.034070 11885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:38.034185 11885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "f9604711050349408ca7da63fe78ca1c"
format_stamp: "Formatted at 2026-08-11 02:37:38 on dist-test-slave-cvwc"
server_key: "7e255b10631b2b762155f4eae1217980"
server_key_iv: "bcc871f74896003d28d49ced654ddb2a"
server_key_version: "encryptionkey@0"
I20260811 02:37:38.034281 11885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:38.055119 11885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:38.055522 11885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:38.055670 11885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:38.060700 11885 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40425
I20260811 02:37:38.060748 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40425 every 8 connection(s)
I20260811 02:37:38.061123 11885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:38.062170 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:38.064618 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c: Bootstrap starting.
I20260811 02:37:38.065217 11994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:38.065474 11994 log.cc:824] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:38.065980 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 11885
I20260811 02:37:38.066085 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:38.066280 3052 external_mini_cluster.cc:1428] Setting key 540f713a4931015c0b7fdec0cb0b53aa
I20260811 02:37:38.066319 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c: No bootstrap required, opened a new log
I20260811 02:37:38.069227 11994 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9604711050349408ca7da63fe78ca1c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40425 } }
I20260811 02:37:38.069461 11994 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:38.069506 11994 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9604711050349408ca7da63fe78ca1c, State: Initialized, Role: FOLLOWER
I20260811 02:37:38.069658 11994 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9604711050349408ca7da63fe78ca1c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40425 } }
I20260811 02:37:38.069788 11994 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:38.069835 11994 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:38.069877 11994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:38.070689 11994 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9604711050349408ca7da63fe78ca1c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40425 } }
I20260811 02:37:38.070809 11994 leader_election.cc:302] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9604711050349408ca7da63fe78ca1c; no voters:
I20260811 02:37:38.070998 11994 leader_election.cc:288] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:38.071087 12007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:38.071259 12007 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [term 1 LEADER]: Becoming Leader. State: Replica: f9604711050349408ca7da63fe78ca1c, State: Running, Role: LEADER
I20260811 02:37:38.071439 12007 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9604711050349408ca7da63fe78ca1c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40425 } }
I20260811 02:37:38.071655 11994 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:38.071823 12008 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9604711050349408ca7da63fe78ca1c. Latest consensus state: current_term: 1 leader_uuid: "f9604711050349408ca7da63fe78ca1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9604711050349408ca7da63fe78ca1c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40425 } } }
I20260811 02:37:38.071900 12008 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:38.072045 12009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9604711050349408ca7da63fe78ca1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9604711050349408ca7da63fe78ca1c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40425 } } }
I20260811 02:37:38.072098 12009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:38.072214 12014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:38.072701 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:38.066838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.2.251.62:40425)
0811 02:37:38.067124 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:38.067132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:38.067158 (+ 26us) server_negotiation.cc:407] Connection header received
0811 02:37:38.067810 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:38.067821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:38.067893 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:38.068025 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:38.068326 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:38.069246 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:38.070398 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:38.070862 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:38.071019 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:38.071766 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:38.071771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:38.071771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:37:38.072014 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:38.072057 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:37:38.072074 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:38.072146 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:38.072425 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 02:37:38.072508 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":145,"threads_started":1}
I20260811 02:37:38.072798 12014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:38.073910 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:37:38.078291 12014 catalog_manager.cc:1347] Generated new cluster ID: 5aae42c98e74494daa4a8ae408b11910
I20260811 02:37:38.078369 12014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:38.117177 12014 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:38.117802 12014 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:38.124861 12014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9604711050349408ca7da63fe78ca1c: Generated new TSK 0
I20260811 02:37:38.125092 12014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:37:38.231447 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:38.137831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.2.251.62:40425)
0811 02:37:38.137963 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:38.137966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:38.138035 (+ 69us) server_negotiation.cc:407] Connection header received
0811 02:37:38.138099 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:38.138101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:38.138119 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:38.138220 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0811 02:37:38.138489 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:38.139257 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:38.140113 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:38.140410 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:38.140495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:38.140732 (+ 237us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 02:37:38.231176 (+ 90444us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 02:37:38.231377 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47926: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":99}
W20260811 02:37:38.231528 12002 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47926: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 02:37:38.231652 12056 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:38.137752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40425 (local address 127.0.0.1:47926)
0811 02:37:38.137958 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:38.137970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:38.138035 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:38.138252 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:38.138254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:38.138256 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0811 02:37:38.138328 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:38.138330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:38.139372 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:38.139373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:38.140010 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:38.140012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:38.140529 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:38.140530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:38.140589 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:38.140596 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 02:37:38.231382 (+ 90786us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 02:37:38.231533 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.251.62:40425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20260811 02:37:38.232056 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 11885
2026-08-11T02:37:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (349 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (113 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (106 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:38 dist-test-slave-cvwc krb5kdc[12075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:38 dist-test-slave-cvwc krb5kdc[12075](info): set up 2 sockets
Aug 11 02:37:38 dist-test-slave-cvwc krb5kdc[12075](info): commencing operation
krb5kdc: starting...
W20260811 02:37:39.705871 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:39 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:39.853897 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:37323
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:39.944533 12096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:39.944804 12096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:39.944842 12096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:39.946270 12096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:39.946332 12096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:39.946352 12096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:39.946368 12096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:39.946383 12096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:39.948146 12096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:37323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:39.948452 12096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:39.948712 12096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:39.954473 12110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:39.954457 12107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:39.954457 12104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:39.954635 12096 server_base.cc:1034] running on GCE node
I20260811 02:37:39.955101 12096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:39.955464 12096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:39.956629 12096 hybrid_clock.cc:648] HybridClock initialized: now 1786415859956594 us; error 57 us; skew 500 ppm
Aug 11 02:37:39 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415859, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:39.959686 12096 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:39.960243 12096 webserver.cc:460] Webserver started at http://127.2.251.62:44237/ using document root <none> and password file <none>
I20260811 02:37:39.960528 12096 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:39.960578 12096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:39.960729 12096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:39.962014 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "bc302102f7af437cb661df7eb4b4b034"
format_stamp: "Formatted at 2026-08-11 02:37:39 on dist-test-slave-cvwc"
server_key: "8967dfaeac6331a2d862f2d0d3f542df"
server_key_iv: "3a014d3b6b6e78021eab851b53b0eaba"
server_key_version: "encryptionkey@0"
I20260811 02:37:39.962405 12096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "bc302102f7af437cb661df7eb4b4b034"
format_stamp: "Formatted at 2026-08-11 02:37:39 on dist-test-slave-cvwc"
server_key: "8967dfaeac6331a2d862f2d0d3f542df"
server_key_iv: "3a014d3b6b6e78021eab851b53b0eaba"
server_key_version: "encryptionkey@0"
I20260811 02:37:39.964059 12096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:39.965054 12121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:39.965281 12096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:39.965384 12096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "bc302102f7af437cb661df7eb4b4b034"
format_stamp: "Formatted at 2026-08-11 02:37:39 on dist-test-slave-cvwc"
server_key: "8967dfaeac6331a2d862f2d0d3f542df"
server_key_iv: "3a014d3b6b6e78021eab851b53b0eaba"
server_key_version: "encryptionkey@0"
I20260811 02:37:39.965456 12096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:39.982563 12096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:39.985525 12096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:39.985703 12096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:39.990185 12096 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:37323
I20260811 02:37:39.990228 12212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:37323 every 8 connection(s)
I20260811 02:37:39.990617 12096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:39.991058 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 12096
I20260811 02:37:39.991161 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:39.991389 3052 external_mini_cluster.cc:1428] Setting key a34df58486491b88f248d8faf9df68f5
I20260811 02:37:39.991830 12215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:39.995265 12215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034: Bootstrap starting.
I20260811 02:37:39.996091 12215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:39.996402 12215 log.cc:824] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:39.997210 12215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034: No bootstrap required, opened a new log
I20260811 02:37:39.999514 12215 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc302102f7af437cb661df7eb4b4b034" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37323 } }
I20260811 02:37:39.999683 12215 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:39.999707 12215 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc302102f7af437cb661df7eb4b4b034, State: Initialized, Role: FOLLOWER
I20260811 02:37:39.999821 12215 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc302102f7af437cb661df7eb4b4b034" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37323 } }
I20260811 02:37:39.999907 12215 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:39.999933 12215 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:39.999969 12215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:40.000613 12215 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc302102f7af437cb661df7eb4b4b034" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37323 } }
I20260811 02:37:40.000725 12215 leader_election.cc:302] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc302102f7af437cb661df7eb4b4b034; no voters:
I20260811 02:37:40.000898 12215 leader_election.cc:288] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:40.000973 12225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:40.001166 12225 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [term 1 LEADER]: Becoming Leader. State: Replica: bc302102f7af437cb661df7eb4b4b034, State: Running, Role: LEADER
I20260811 02:37:40.001206 12215 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:40.001292 12225 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc302102f7af437cb661df7eb4b4b034" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37323 } }
I20260811 02:37:40.001765 12228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc302102f7af437cb661df7eb4b4b034. Latest consensus state: current_term: 1 leader_uuid: "bc302102f7af437cb661df7eb4b4b034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc302102f7af437cb661df7eb4b4b034" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37323 } } }
I20260811 02:37:40.001878 12226 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc302102f7af437cb661df7eb4b4b034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc302102f7af437cb661df7eb4b4b034" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 37323 } } }
I20260811 02:37:40.001981 12226 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:40.001931 12228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034 [sys.catalog]: This master's current role is: LEADER
Aug 11 02:37:39 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
W20260811 02:37:40.003171 12248 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:37:40.003819 12248 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:37:40.003909 12238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:40.004159 12238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:40.006919 12238 catalog_manager.cc:1347] Generated new cluster ID: 1e724207e92f4d838ef09bfe89f2f95e
I20260811 02:37:40.007009 12238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:40.009368 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:39.992037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.2.251.62:37323)
0811 02:37:39.992255 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:39.992265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:39.992299 (+ 34us) server_negotiation.cc:407] Connection header received
0811 02:37:39.993002 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:39.993013 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:39.993087 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:39.993265 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:39.994672 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:39.996032 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:39.996964 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:39.997589 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:39.997753 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.004314 (+ 6561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.004319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.004319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.004465 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.007017 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.007700 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.007707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.007708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.007902 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.008170 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.008171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.008171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.008616 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.008802 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.008927 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.009136 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260811 02:37:40.020901 12238 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:40.021549 12238 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:40.028344 12238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc302102f7af437cb661df7eb4b4b034: Generated new TSK 0
I20260811 02:37:40.028704 12238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:40.065375 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:37323
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:40.170516 12255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:40.170825 12255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:40.170858 12255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:40.172904 12255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:40.172986 12255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:40.173020 12255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:40.175099 12255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:40.175383 12255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:40.175609 12255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:40.182446 12265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:40.182464 12264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:40.182483 12269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:40.182842 12255 server_base.cc:1034] running on GCE node
I20260811 02:37:40.183269 12255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:40.184089 12255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:40.185357 12255 hybrid_clock.cc:648] HybridClock initialized: now 1786415860185345 us; error 48 us; skew 500 ppm
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.189957 12255 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:40.190594 12255 webserver.cc:460] Webserver started at http://127.2.251.1:34623/ using document root <none> and password file <none>
I20260811 02:37:40.190866 12255 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:40.190922 12255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:40.191061 12255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:40.192307 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "5c976552abf04d968012b2817e0e9d05"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "8317882f2a012f16be89d2201dacc523"
server_key_iv: "92f025408599a32a0d2609a3480b3229"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.192673 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "5c976552abf04d968012b2817e0e9d05"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "8317882f2a012f16be89d2201dacc523"
server_key_iv: "92f025408599a32a0d2609a3480b3229"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.194430 12255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260811 02:37:40.195539 12285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.195874 12255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:40.196056 12255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "5c976552abf04d968012b2817e0e9d05"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "8317882f2a012f16be89d2201dacc523"
server_key_iv: "92f025408599a32a0d2609a3480b3229"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.196162 12255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:40.208666 12255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:40.211938 12255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:40.212100 12255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:40.212342 12255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:40.212667 12255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:40.212698 12255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.212721 12255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:40.212733 12255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.219182 12255 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:43629
I20260811 02:37:40.219249 12497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:43629 every 8 connection(s)
I20260811 02:37:40.219633 12255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:40.224334 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 12255
I20260811 02:37:40.224442 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:40.224638 3052 external_mini_cluster.cc:1428] Setting key a93da205002b053c94a3f80a3786ef09
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:40.230880 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.220479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:42061 (local address 127.2.251.62:37323)
0811 02:37:40.220586 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.220589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.220847 (+ 258us) server_negotiation.cc:407] Connection header received
0811 02:37:40.221844 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.221846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.221858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.221916 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.222587 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.223129 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.224298 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.224569 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.224633 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.227683 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.227686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.227686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.227697 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.229344 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.229830 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.229832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.229832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.229900 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.230157 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.230158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.230159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.230285 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.230342 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.230661 (+ 319us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.230774 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 02:37:40.231050 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.220439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:37323 (local address 127.2.251.1:42061)
0811 02:37:40.220699 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:40.220742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:40.221699 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:40.221991 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:40.221995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:40.222275 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:40.222478 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:40.222482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.223255 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.223256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:40.224181 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:40.224184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.224651 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.224652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:40.224810 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.225664 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:40.225676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:40.227556 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:40.229451 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:40.229462 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:40.229465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:40.229762 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:40.229966 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:40.229967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:40.229968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:40.230089 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:40.230342 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:40.230350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:40.230521 (+ 171us) client_negotiation.cc:769] Sending connection context
0811 02:37:40.230616 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 02:37:40.230803 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":143,"threads_started":1}
I20260811 02:37:40.231613 12499 heartbeater.cc:347] Connected to a master server at 127.2.251.62:37323
I20260811 02:37:40.231735 12499 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:40.231957 12499 heartbeater.cc:511] Master 127.2.251.62:37323 requested a full tablet report, sending...
I20260811 02:37:40.232535 12146 ts_manager.cc:194] Registered new tserver with Master: 5c976552abf04d968012b2817e0e9d05 (127.2.251.1:43629)
I20260811 02:37:40.233170 12146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:42061
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:40.258316 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:37323
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:40.362215 12508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:40.362588 12508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:40.362633 12508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:40.364507 12508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:40.364571 12508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:40.364603 12508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:40.366719 12508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:40.367035 12508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:40.367301 12508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:40.372994 12523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:40.373037 12521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:40.373037 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:40.373281 12508 server_base.cc:1034] running on GCE node
I20260811 02:37:40.373602 12508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:40.373898 12508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:40.375141 12508 hybrid_clock.cc:648] HybridClock initialized: now 1786415860375118 us; error 31 us; skew 500 ppm
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.378865 12508 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:40.379673 12508 webserver.cc:460] Webserver started at http://127.2.251.2:38685/ using document root <none> and password file <none>
I20260811 02:37:40.379947 12508 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:40.380005 12508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:40.380169 12508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:40.381494 12508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "914c7e970fc549c7ab45ad0d01fd3a27"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "ffec96aa37570ab829ce8e589a784393"
server_key_iv: "e7fa3a74d34797a707989f3e57f12520"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.381963 12508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "914c7e970fc549c7ab45ad0d01fd3a27"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "ffec96aa37570ab829ce8e589a784393"
server_key_iv: "e7fa3a74d34797a707989f3e57f12520"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.383867 12508 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:40.385056 12535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.385401 12508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:40.385530 12508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "914c7e970fc549c7ab45ad0d01fd3a27"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "ffec96aa37570ab829ce8e589a784393"
server_key_iv: "e7fa3a74d34797a707989f3e57f12520"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.385618 12508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:40.394958 12508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:40.398264 12508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:40.398469 12508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:40.398751 12508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:40.399206 12508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:40.399271 12508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.399324 12508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:40.399343 12508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.408083 12508 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38119
I20260811 02:37:40.408180 12752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38119 every 8 connection(s)
I20260811 02:37:40.408535 12508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:40.417164 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 12508
I20260811 02:37:40.417286 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:40.417519 3052 external_mini_cluster.cc:1428] Setting key d5c6bc801d7d209203e4a472b05269b9
I20260811 02:37:40.420984 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.409628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:57315 (local address 127.2.251.62:37323)
0811 02:37:40.409755 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.409759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.410072 (+ 313us) server_negotiation.cc:407] Connection header received
0811 02:37:40.411127 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.411128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.411142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.411214 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.412088 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.412620 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.413775 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.414009 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.414076 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.416813 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.416817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.416817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.416829 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.418966 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.419688 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.419690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.419690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.419782 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.420107 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.420108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.420108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.420266 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.420330 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.420724 (+ 394us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.420861 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 02:37:40.421125 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.409511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:37323 (local address 127.2.251.2:57315)
0811 02:37:40.409876 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:40.409926 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:40.410971 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:40.411392 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:40.411397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:40.411714 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:40.411965 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:40.411970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.412721 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.412722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:40.413621 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:40.413623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.414121 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.414122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:40.414257 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.415013 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:40.415023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:40.416618 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:40.419134 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:40.419148 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:40.419152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:40.419536 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:40.419859 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:40.419860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:40.419860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:40.420026 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:40.420332 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:40.420340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:40.420540 (+ 200us) client_negotiation.cc:769] Sending connection context
0811 02:37:40.420664 (+ 124us) client_negotiation.cc:240] Negotiation successful
0811 02:37:40.420903 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
I20260811 02:37:40.421798 12753 heartbeater.cc:347] Connected to a master server at 127.2.251.62:37323
I20260811 02:37:40.421919 12753 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:40.422125 12753 heartbeater.cc:511] Master 127.2.251.62:37323 requested a full tablet report, sending...
I20260811 02:37:40.422585 12146 ts_manager.cc:194] Registered new tserver with Master: 914c7e970fc549c7ab45ad0d01fd3a27 (127.2.251.2:38119)
I20260811 02:37:40.423091 12146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:57315
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:40.452991 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:37323
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:40.545344 12759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:40.545603 12759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:40.545639 12759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:40.547000 12759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:40.547070 12759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:40.547109 12759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:40.548830 12759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:40.549140 12759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:40.549400 12759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:40.556160 12776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:40.556047 12773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:40.556044 12774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:40.556499 12759 server_base.cc:1034] running on GCE node
I20260811 02:37:40.556749 12759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:40.557044 12759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:40.558224 12759 hybrid_clock.cc:648] HybridClock initialized: now 1786415860558204 us; error 38 us; skew 500 ppm
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.561193 12759 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:40.561803 12759 webserver.cc:460] Webserver started at http://127.2.251.3:32785/ using document root <none> and password file <none>
I20260811 02:37:40.562041 12759 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:40.562098 12759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:40.562211 12759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:40.563401 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "664f1d3b39364555a480f31c61ff8b27"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "b334af72880ad456f25b7bcbc27a3004"
server_key_iv: "95cb8c1b1193d4372c298876cc7cb7df"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.563755 12759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "664f1d3b39364555a480f31c61ff8b27"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "b334af72880ad456f25b7bcbc27a3004"
server_key_iv: "95cb8c1b1193d4372c298876cc7cb7df"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.565240 12759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:37:40.566346 12791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.566666 12759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:40.566776 12759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "664f1d3b39364555a480f31c61ff8b27"
format_stamp: "Formatted at 2026-08-11 02:37:40 on dist-test-slave-cvwc"
server_key: "b334af72880ad456f25b7bcbc27a3004"
server_key_iv: "95cb8c1b1193d4372c298876cc7cb7df"
server_key_version: "encryptionkey@0"
I20260811 02:37:40.566866 12759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:40.591400 12759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:40.594481 12759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:40.594784 12759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:40.595320 12759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:40.595820 12759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:40.595862 12759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.595887 12759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:40.595899 12759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:40.603305 12759 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:33279
I20260811 02:37:40.603449 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:33279 every 8 connection(s)
I20260811 02:37:40.603788 12759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:40.610987 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 12759
I20260811 02:37:40.611100 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:40.611274 3052 external_mini_cluster.cc:1428] Setting key 991e8558a220fe7cd87151e1e8501a2e
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:40.617023 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.604740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:55375 (local address 127.2.251.62:37323)
0811 02:37:40.604807 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.604810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.605314 (+ 504us) server_negotiation.cc:407] Connection header received
0811 02:37:40.606610 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.606612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.606631 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.606694 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.607519 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.608102 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.609212 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.609426 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.609488 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.612769 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.612771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.612771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.612783 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.614948 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.615690 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.615692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.615692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.615781 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.616249 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.616250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.616250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.616431 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.616509 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.616835 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.616938 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 02:37:40.617282 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.604714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:37323 (local address 127.2.251.3:55375)
0811 02:37:40.605084 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:40.605143 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:40.606439 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:40.606793 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:40.606799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:40.607161 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:40.607419 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:40.607424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.608195 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.608196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:40.609122 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:40.609125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.609512 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.609513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:40.609651 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.610402 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:40.610414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:40.612640 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:40.615084 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:40.615097 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:40.615101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:40.615449 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:40.615890 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:40.615891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:40.615892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:40.616124 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:40.616510 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:40.616516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:40.616706 (+ 190us) client_negotiation.cc:769] Sending connection context
0811 02:37:40.616811 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 02:37:40.617010 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":194,"threads_started":1}
I20260811 02:37:40.617857 12988 heartbeater.cc:347] Connected to a master server at 127.2.251.62:37323
I20260811 02:37:40.618093 12988 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:40.618501 12988 heartbeater.cc:511] Master 127.2.251.62:37323 requested a full tablet report, sending...
I20260811 02:37:40.619740 12146 ts_manager.cc:194] Registered new tserver with Master: 664f1d3b39364555a480f31c61ff8b27 (127.2.251.3:33279)
I20260811 02:37:40.620374 12146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:55375
I20260811 02:37:40.624562 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/test-admin.keytab.
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.648804 3052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 02:37:40.651598 12999 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:40.650327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.2.251.1:43629)
0811 02:37:40.650520 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.650526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.650540 (+ 14us) server_negotiation.cc:407] Connection header received
0811 02:37:40.650574 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.650576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.650593 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.650684 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.651469 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46680: BlockingRecv error: recv got EOF from 127.0.0.1:46680 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
W20260811 02:37:40.651698 3052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc' of type FILE
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.653383 3052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.654749 3052 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:40.665256 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.655505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.2.251.1:43629)
0811 02:37:40.655593 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.655595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.655608 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:40.655641 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.655642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.655656 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.655717 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.656180 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.656959 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.657944 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.658375 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.658452 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.661296 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.661312 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.661312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.661323 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.663475 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.663976 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.663979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.663980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.664050 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.664385 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.664386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.664387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.664649 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.664801 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.665060 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.665155 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 02:37:40.667835 12999 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:40.667168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.2.251.1:43629)
0811 02:37:40.667258 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.667260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.667270 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:40.667348 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.667349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.667361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.667451 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.667787 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46706: BlockingRecv error: recv got EOF from 127.0.0.1:46706 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260811 02:37:40.668068 3052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc' of type FILE
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.669466 3052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.670449 3052 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:40.679291 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.671085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.2.251.1:43629)
0811 02:37:40.671142 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.671145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.671234 (+ 89us) server_negotiation.cc:407] Connection header received
0811 02:37:40.671326 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.671327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.671340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.671380 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.671803 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.672323 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.673222 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.673463 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.673534 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.676134 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.676136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.676137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.676146 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.677613 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.678204 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.678206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.678206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.678289 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.678664 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.678665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.678665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.678791 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.678887 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.679085 (+ 198us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.679179 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260811 02:37:40.681349 12999 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:40.680641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.2.251.1:43629)
0811 02:37:40.680683 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.680685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.680695 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:40.680743 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.680744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.680754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.680834 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.681292 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46712: BlockingRecv error: recv got EOF from 127.0.0.1:46712 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260811 02:37:40.681500 3052 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786415831315317-3052-0/krb5kdc/krb5cc)
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.682663 3052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:40.683529 3052 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[12075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:37:40.692242 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:40.684119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.2.251.1:43629)
0811 02:37:40.684179 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:40.684181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:40.684236 (+ 55us) server_negotiation.cc:407] Connection header received
0811 02:37:40.684287 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:40.684288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:40.684298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:40.684348 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:40.685127 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.685898 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.686708 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:40.686930 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:40.687021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:40.689406 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:40.689408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:40.689408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:40.689418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:40.691084 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.691484 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.691485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.691485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.691548 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:40.691742 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:40.691743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:40.691743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:40.691852 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:40.691934 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:40.692100 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 02:37:40.692168 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:37:40.692960 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 12255
I20260811 02:37:40.710958 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 12508
I20260811 02:37:40.725445 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 12759
I20260811 02:37:40.742211 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 12096
2026-08-11T02:37:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2295 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[13021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[13021](info): set up 2 sockets
Aug 11 02:37:40 dist-test-slave-cvwc krb5kdc[13021](info): commencing operation
krb5kdc: starting...
W20260811 02:37:41.991535 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab.
Entry for principal oryx/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab.
I20260811 02:37:42.146638 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:41783
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab
--principal=oryx/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:42.252650 13048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:42.252942 13048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:42.252981 13048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:42.254424 13048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:42.254480 13048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:42.254500 13048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:42.254516 13048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:42.254531 13048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:42.256307 13048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab
--principal=oryx/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:42.256614 13048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:42.256860 13048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:42.262492 13059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.262485 13061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.262742 13064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:42.262863 13048 server_base.cc:1034] running on GCE node
I20260811 02:37:42.263149 13048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:42.263481 13048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:42.264679 13048 hybrid_clock.cc:648] HybridClock initialized: now 1786415862264635 us; error 60 us; skew 500 ppm
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:42.267851 13048 init.cc:377] Logged in from keytab as oryx/127.2.251.62@KRBTEST.COM (short username oryx)
I20260811 02:37:42.268383 13048 webserver.cc:460] Webserver started at http://127.2.251.62:44603/ using document root <none> and password file <none>
I20260811 02:37:42.268620 13048 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:42.268675 13048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:42.268812 13048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:42.270571 13048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "c7c088fc74e3491fb0b81d75400b7332"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "8efb6c0861bdc2804a41377f57f88acd"
server_key_iv: "9c9a189936250bd8e43fb2d12de12e0c"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.270979 13048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "c7c088fc74e3491fb0b81d75400b7332"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "8efb6c0861bdc2804a41377f57f88acd"
server_key_iv: "9c9a189936250bd8e43fb2d12de12e0c"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.272590 13048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:37:42.273747 13078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.273979 13048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:42.274114 13048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "c7c088fc74e3491fb0b81d75400b7332"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "8efb6c0861bdc2804a41377f57f88acd"
server_key_iv: "9c9a189936250bd8e43fb2d12de12e0c"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.274192 13048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:42.291236 13048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:42.294715 13048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:42.295140 13048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:42.299880 13048 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:41783
I20260811 02:37:42.300118 13161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:41783 every 8 connection(s)
I20260811 02:37:42.300346 13048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:42.301457 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:42.304504 13162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Bootstrap starting.
I20260811 02:37:42.305254 13162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:42.305568 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 13048
I20260811 02:37:42.305598 13162 log.cc:824] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:42.305680 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:42.305941 3052 external_mini_cluster.cc:1428] Setting key a4d146224b97e8aa606b1d557dd2a0e7
I20260811 02:37:42.306562 13162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: No bootstrap required, opened a new log
I20260811 02:37:42.308728 13162 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } }
I20260811 02:37:42.308907 13162 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:42.308930 13162 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c088fc74e3491fb0b81d75400b7332, State: Initialized, Role: FOLLOWER
I20260811 02:37:42.309063 13162 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } }
I20260811 02:37:42.309154 13162 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:42.309193 13162 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:42.309237 13162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:42.310487 13162 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } }
I20260811 02:37:42.310597 13162 leader_election.cc:302] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7c088fc74e3491fb0b81d75400b7332; no voters:
I20260811 02:37:42.310772 13162 leader_election.cc:288] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:42.310827 13168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:42.311038 13168 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 LEADER]: Becoming Leader. State: Replica: c7c088fc74e3491fb0b81d75400b7332, State: Running, Role: LEADER
I20260811 02:37:42.311077 13162 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:42.311221 13168 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } }
I20260811 02:37:42.311650 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } }
I20260811 02:37:42.311730 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:42.311955 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7c088fc74e3491fb0b81d75400b7332. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } }
I20260811 02:37:42.312439 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:42.312538 13189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:42.313222 13189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:42.314965 13189 catalog_manager.cc:1347] Generated new cluster ID: fa29a9b4fca3464db553767f6a8aa3e7
I20260811 02:37:42.315029 13189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:42.318799 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.306531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.2.251.62:41783)
0811 02:37:42.306764 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:42.306773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:42.306801 (+ 28us) server_negotiation.cc:407] Connection header received
0811 02:37:42.308729 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:42.308741 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:42.308821 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:42.309060 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:42.309877 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.310922 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.312036 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.312668 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.312816 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.315607 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:42.315612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:42.315613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:42.315627 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:42.317240 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.317586 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.317589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.317589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.317703 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.317910 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.317911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.317911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.318187 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:42.318339 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:42.318446 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 02:37:42.318618 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260811 02:37:42.330677 13189 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:42.331594 13189 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:42.337488 13189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Generated new TSK 0
I20260811 02:37:42.337749 13189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab.
Entry for principal oryx/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab.
I20260811 02:37:42.378221 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41783
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:42.478739 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:42.479039 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:42.479071 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:42.480492 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:42.480553 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:42.480585 13206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:42.482323 13206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:42.482589 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:42.482857 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:42.489892 13215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.490027 13214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.490029 13218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:42.490067 13206 server_base.cc:1034] running on GCE node
I20260811 02:37:42.490427 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:42.490691 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:42.491881 13206 hybrid_clock.cc:648] HybridClock initialized: now 1786415862491852 us; error 39 us; skew 500 ppm
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:42.495236 13206 init.cc:377] Logged in from keytab as oryx/127.2.251.1@KRBTEST.COM (short username oryx)
I20260811 02:37:42.495810 13206 webserver.cc:460] Webserver started at http://127.2.251.1:44453/ using document root <none> and password file <none>
I20260811 02:37:42.496093 13206 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:42.496145 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:42.496362 13206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:42.497499 13206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "0e29543d60f5459bb151b5cfcb4e64dc"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "3bf723bdec5be672ac02d0d55dd4e93f"
server_key_iv: "3bbdbdf59fe0f3d73c8550b9ed223fc2"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.497889 13206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "0e29543d60f5459bb151b5cfcb4e64dc"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "3bf723bdec5be672ac02d0d55dd4e93f"
server_key_iv: "3bbdbdf59fe0f3d73c8550b9ed223fc2"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.499449 13206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:42.500594 13234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.500954 13206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:37:42.501073 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "0e29543d60f5459bb151b5cfcb4e64dc"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "3bf723bdec5be672ac02d0d55dd4e93f"
server_key_iv: "3bbdbdf59fe0f3d73c8550b9ed223fc2"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.501156 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:42.520651 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:42.524387 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:42.524731 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:42.525408 13206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:42.526360 13206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:42.526422 13206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.526458 13206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:42.526475 13206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.532833 13206 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:34981
I20260811 02:37:42.532881 13407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:34981 every 8 connection(s)
I20260811 02:37:42.533223 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:42.536331 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 13206
I20260811 02:37:42.536423 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:42.536628 3052 external_mini_cluster.cc:1428] Setting key 11dd0997c671cc588628faff77fec315
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:42.544677 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.534160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:47927 (local address 127.2.251.62:41783)
0811 02:37:42.534233 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:42.534235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:42.534372 (+ 137us) server_negotiation.cc:407] Connection header received
0811 02:37:42.534910 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:42.534912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:42.534929 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:42.534982 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:42.536002 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.536570 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.537777 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.538009 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.538070 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.540828 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:42.540831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:42.540832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:42.540845 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:42.542605 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.543463 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.543464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.543465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.543541 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.543883 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.543884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.543884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.544044 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:42.544124 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:42.544466 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 02:37:42.544573 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 02:37:42.544827 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.534045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:47927)
0811 02:37:42.534225 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:42.534276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:42.534797 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:42.535057 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:42.535062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:42.535669 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:42.535896 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:42.535901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.536659 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.536660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:42.537682 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:42.537686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.538089 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.538090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:42.538231 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.538943 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:42.538956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:42.540697 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:42.542785 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:42.542803 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:42.542807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:42.543365 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:42.543662 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:42.543663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:42.543664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:42.543807 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:42.544145 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:42.544153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:42.544310 (+ 157us) client_negotiation.cc:769] Sending connection context
0811 02:37:42.544433 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 02:37:42.544593 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260811 02:37:42.545485 13408 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41783
I20260811 02:37:42.545589 13408 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:42.545819 13408 heartbeater.cc:511] Master 127.2.251.62:41783 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:37:42.546515 13105 ts_manager.cc:194] Registered new tserver with Master: 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981)
I20260811 02:37:42.547168 13105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:47927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab.
Entry for principal oryx/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab.
I20260811 02:37:42.570843 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41783
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:42.675140 13414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:42.675390 13414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:42.675418 13414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:42.676699 13414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:42.676762 13414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:42.676792 13414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:42.679193 13414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:42.679632 13414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:42.679963 13414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:42.686803 13422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.686836 13421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:42.686991 13414 server_base.cc:1034] running on GCE node
W20260811 02:37:42.687076 13424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:42.687383 13414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:42.687673 13414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:42.688861 13414 hybrid_clock.cc:648] HybridClock initialized: now 1786415862688839 us; error 33 us; skew 500 ppm
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:42.691715 13414 init.cc:377] Logged in from keytab as oryx/127.2.251.2@KRBTEST.COM (short username oryx)
I20260811 02:37:42.692221 13414 webserver.cc:460] Webserver started at http://127.2.251.2:38511/ using document root <none> and password file <none>
I20260811 02:37:42.692463 13414 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:42.692518 13414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:42.692654 13414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:42.693966 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "e039f61b46234dc88ed53dc615dfb572"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "1e42cc45f36b940cac51953a3a683f44"
server_key_iv: "e709b6294a36a684fea4d457040e4560"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.694393 13414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "e039f61b46234dc88ed53dc615dfb572"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "1e42cc45f36b940cac51953a3a683f44"
server_key_iv: "e709b6294a36a684fea4d457040e4560"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.695880 13414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:42.696775 13438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.696978 13414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:42.697085 13414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "e039f61b46234dc88ed53dc615dfb572"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "1e42cc45f36b940cac51953a3a683f44"
server_key_iv: "e709b6294a36a684fea4d457040e4560"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.697152 13414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:42.709332 13414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:42.712761 13414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:42.712921 13414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:42.713179 13414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:42.713523 13414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:42.713559 13414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.713589 13414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:42.713613 13414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.720594 13414 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:43099
I20260811 02:37:42.720646 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:43099 every 8 connection(s)
I20260811 02:37:42.721009 13414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:42.728456 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 13414
I20260811 02:37:42.728595 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:42.729015 3052 external_mini_cluster.cc:1428] Setting key 3468e66fd941be26867bbf101042156e
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:42.735581 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.721837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:36051 (local address 127.2.251.62:41783)
0811 02:37:42.721911 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:42.721914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:42.722211 (+ 297us) server_negotiation.cc:407] Connection header received
0811 02:37:42.722735 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:42.722736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:42.722753 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:42.722814 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:42.724061 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.724688 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.726920 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.727267 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.727342 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.731172 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:42.731174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:42.731175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:42.731185 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:42.733223 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.734038 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.734040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.734040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.734117 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.734480 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.734481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.734482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.734620 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:42.734682 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:42.735317 (+ 635us) server_negotiation.cc:299] Negotiation successful
0811 02:37:42.735428 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 02:37:42.736007 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.721817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:36051)
0811 02:37:42.722020 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:42.722065 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:42.722596 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:42.722936 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:42.722945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:42.723567 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:42.723789 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:42.723794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.724812 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.724813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:42.725751 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:42.725756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.727476 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.727480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:42.727718 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.728609 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:42.728633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:42.731046 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:42.733379 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:42.733401 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:42.733407 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:42.733896 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:42.734201 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:42.734201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:42.734202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:42.734385 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:42.734758 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:42.734769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:42.735031 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 02:37:42.735230 (+ 199us) client_negotiation.cc:240] Negotiation successful
0811 02:37:42.735492 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260811 02:37:42.736822 13627 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41783
I20260811 02:37:42.736958 13627 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:42.737217 13627 heartbeater.cc:511] Master 127.2.251.62:41783 requested a full tablet report, sending...
I20260811 02:37:42.737703 13105 ts_manager.cc:194] Registered new tserver with Master: e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099)
I20260811 02:37:42.738138 13105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:36051
WARNING: no policy specified for oryx/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab.
Entry for principal oryx/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab.
I20260811 02:37:42.765688 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41783
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:42.876926 13635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:42.877180 13635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:42.877207 13635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:42.878629 13635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:42.878691 13635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:42.878722 13635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:42.880632 13635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:42.880909 13635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:42.881317 13635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:42.887598 13653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.887650 13654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:42.887598 13658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:42.888150 13635 server_base.cc:1034] running on GCE node
I20260811 02:37:42.888429 13635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:42.888805 13635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:42.890002 13635 hybrid_clock.cc:648] HybridClock initialized: now 1786415862889962 us; error 53 us; skew 500 ppm
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:42.892851 13635 init.cc:377] Logged in from keytab as oryx/127.2.251.3@KRBTEST.COM (short username oryx)
I20260811 02:37:42.893328 13635 webserver.cc:460] Webserver started at http://127.2.251.3:41257/ using document root <none> and password file <none>
I20260811 02:37:42.893558 13635 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:42.893610 13635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:42.893766 13635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:42.894991 13635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "cfc87615c43047858efb37206c32fe1f"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "6b6fd97d5fe1cc824c94ff222025a3d4"
server_key_iv: "903cd530b28b3475e46583ab8bd30ef6"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.895383 13635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "cfc87615c43047858efb37206c32fe1f"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "6b6fd97d5fe1cc824c94ff222025a3d4"
server_key_iv: "903cd530b28b3475e46583ab8bd30ef6"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.897078 13635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:37:42.898160 13669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.898448 13635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:42.898561 13635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "cfc87615c43047858efb37206c32fe1f"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "6b6fd97d5fe1cc824c94ff222025a3d4"
server_key_iv: "903cd530b28b3475e46583ab8bd30ef6"
server_key_version: "encryptionkey@0"
I20260811 02:37:42.898635 13635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:42.909940 13635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:42.913195 13635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:42.913383 13635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:42.913650 13635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:42.914178 13635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:42.914223 13635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.914258 13635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:42.914275 13635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:42.920902 13635 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:41133
I20260811 02:37:42.920945 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:41133 every 8 connection(s)
I20260811 02:37:42.921304 13635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:42.925148 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 13635
I20260811 02:37:42.925282 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:42.925484 3052 external_mini_cluster.cc:1428] Setting key 4145f35775cbe6a866bed5080a0f89fe
Aug 11 02:37:42 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:42.934803 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.922476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:60041)
0811 02:37:42.922750 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:42.922816 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:42.923549 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:42.923889 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:42.923894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:42.924691 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:42.925050 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:42.925059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.926111 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.926112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:42.927597 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:42.927605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.928104 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.928105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:42.928259 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.929058 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:42.929069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:42.930837 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:42.932730 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:42.932742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:42.932745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:42.933027 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:42.933318 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:42.933319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:42.933319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:42.933456 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:42.934068 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:42.934077 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:42.934266 (+ 189us) client_negotiation.cc:769] Sending connection context
0811 02:37:42.934381 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 02:37:42.934579 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
I20260811 02:37:42.935057 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:42.922534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:60041 (local address 127.2.251.62:41783)
0811 02:37:42.922611 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:42.922613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:42.922964 (+ 351us) server_negotiation.cc:407] Connection header received
0811 02:37:42.923690 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:42.923692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:42.923705 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:42.923791 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:42.925234 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.925938 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.927738 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:42.927964 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:42.928051 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:42.931012 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:42.931015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:42.931015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:42.931029 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:42.932587 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.933135 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.933136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.933137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.933207 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:42.933641 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:42.933644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:42.933645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:42.933933 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:42.934055 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:42.934617 (+ 562us) server_negotiation.cc:299] Negotiation successful
0811 02:37:42.934747 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:37:42.935575 13876 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41783
I20260811 02:37:42.935695 13876 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:42.935902 13876 heartbeater.cc:511] Master 127.2.251.62:41783 requested a full tablet report, sending...
I20260811 02:37:42.936537 13105 ts_manager.cc:194] Registered new tserver with Master: cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133)
I20260811 02:37:42.936990 13105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:60041
I20260811 02:37:42.938434 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.251.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.61.keytab.
Entry for principal oryx/127.2.251.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.61.keytab.
I20260811 02:37:42.973995 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:43011
--webserver_interface=127.2.251.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.61.keytab
--principal=oryx/127.2.251.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:41783,127.2.251.61:43011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:43.075454 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:43.075711 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:43.075737 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:43.077042 13889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:43.077099 13889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:43.077113 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:43.077124 13889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:43.077135 13889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:43.079221 13889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:41783,127.2.251.61:43011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.61.keytab
--principal=oryx/127.2.251.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:43011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.251.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
W20260811 02:37:43.079387 13889 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.251.62:41783,127.2.251.61:43011'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 02:37:43.079555 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:43.079823 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:43.085503 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.085505 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.085805 13899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.087110 13889 server_base.cc:1034] running on GCE node
I20260811 02:37:43.087440 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:43.087809 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:43.089054 13889 hybrid_clock.cc:648] HybridClock initialized: now 1786415863089025 us; error 44 us; skew 500 ppm
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:43.092819 13889 init.cc:377] Logged in from keytab as oryx/127.2.251.61@KRBTEST.COM (short username oryx)
I20260811 02:37:43.094195 13889 webserver.cc:460] Webserver started at http://127.2.251.61:41853/ using document root <none> and password file <none>
I20260811 02:37:43.094563 13889 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.094632 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.094970 13889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:43.096218 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/instance:
uuid: "a7126980727f4079a8447531f08e8269"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "d01c546eaed4804ed279ffb99eb14f2a"
server_key_iv: "4690c17d4fbbe910f5656b04305addfc"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.096777 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal/instance:
uuid: "a7126980727f4079a8447531f08e8269"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "d01c546eaed4804ed279ffb99eb14f2a"
server_key_iv: "4690c17d4fbbe910f5656b04305addfc"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.099133 13889 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 02:37:43.100461 13916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.100807 13889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:43.100947 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
uuid: "a7126980727f4079a8447531f08e8269"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "d01c546eaed4804ed279ffb99eb14f2a"
server_key_iv: "4690c17d4fbbe910f5656b04305addfc"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.101082 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.113467 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:43.116561 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:43.116726 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:43.121816 13889 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.61:43011
I20260811 02:37:43.122032 14000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.61:43011 every 8 connection(s)
I20260811 02:37:43.122292 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
I20260811 02:37:43.123574 14003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:43.124178 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 13889
I20260811 02:37:43.126214 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.61@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:43.135983 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.127371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.2.251.62:41783)
0811 02:37:43.127445 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.127448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.127684 (+ 236us) server_negotiation.cc:407] Connection header received
0811 02:37:43.127755 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.127756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.127768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.127832 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.128345 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.128839 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.129703 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.130021 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.130098 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.132358 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.132360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.132361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.132370 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.134131 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.134864 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.134866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.134866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.134946 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.135194 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.135195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.135195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.135440 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.135520 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.135786 (+ 266us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.135879 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 02:37:43.136196 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.125093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.2.251.61:43011)
0811 02:37:43.125329 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.125339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.125369 (+ 30us) server_negotiation.cc:407] Connection header received
0811 02:37:43.125856 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.125871 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.125929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.126068 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.126933 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.128038 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.128776 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.129226 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.129329 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.131988 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.131996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.131996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.132004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.133931 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.134739 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.134743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.134743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.134862 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.135170 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.135170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.135171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.135503 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.135652 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.135787 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.135966 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
I20260811 02:37:43.136297 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.127312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:41754)
0811 02:37:43.127584 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.127621 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.127701 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.127854 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.127856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.128179 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.128253 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.128255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.128948 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.128949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.129604 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.129607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.130137 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.130138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.130213 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.130713 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.130722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.132217 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.134264 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.134286 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.134290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.134776 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.135010 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.135010 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.135011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.135134 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.135536 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.135545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.135635 (+ 90us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.135756 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.136150 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":118,"thread_start_us":116,"threads_started":1}
I20260811 02:37:43.137033 14003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.61@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
I20260811 02:37:43.144625 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.137551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.2.251.61:43011)
0811 02:37:43.137626 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.137628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.137693 (+ 65us) server_negotiation.cc:407] Connection header received
0811 02:37:43.137789 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.137791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.137805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.137865 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.138159 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.138654 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.139275 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.139504 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.139560 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.141358 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.141359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.141360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.141367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.143411 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.143883 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.143884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.143884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.143950 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.144119 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.144120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.144120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.144240 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.144296 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.144426 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.144513 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 02:37:43.144683 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.137335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58690)
0811 02:37:43.137458 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.137478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.137575 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.137880 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.137881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.138029 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.138086 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.138088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.138723 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.138724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.139202 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.139204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.139564 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.139565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.139624 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.140043 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.140048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.141273 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.143514 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.143515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.143516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.143799 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.144006 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.144007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.144007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.144067 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.144298 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.144299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.144338 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.144397 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.144574 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":76}
I20260811 02:37:43.146876 14003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Bootstrap starting.
I20260811 02:37:43.147847 14003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:43.148447 14003 log.cc:824] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:43.149590 14003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: No bootstrap required, opened a new log
I20260811 02:37:43.151266 14003 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } }
I20260811 02:37:43.151456 14003 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:43.151481 14003 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7126980727f4079a8447531f08e8269, State: Initialized, Role: FOLLOWER
I20260811 02:37:43.151615 14003 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } }
I20260811 02:37:43.152012 14003 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:43.151937 14019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } } }
I20260811 02:37:43.152081 14019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 02:37:43.154505 14035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:37:43.154573 14035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:37:43.158401 13889 master_runner.cc:428] Detected that this master a7126980727f4079a8447531f08e8269 is joining an existing cluster
I20260811 02:37:43.158535 13889 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.251.61:43011
I20260811 02:37:43.167511 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.159745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.2.251.61:43011)
0811 02:37:43.159861 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.159864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.160387 (+ 523us) server_negotiation.cc:407] Connection header received
0811 02:37:43.160485 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.160487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.160496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.160522 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.161095 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.161579 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.162411 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.162620 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.162685 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.164535 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.164536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.164536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.164544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.166133 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.166723 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.166724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.166724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.166778 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.166962 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.166962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.166963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.167076 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.167134 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.167262 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.167387 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":88}
I20260811 02:37:43.167604 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.159693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58696)
0811 02:37:43.160302 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.160313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.160417 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.160572 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.160572 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.160881 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.161016 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.161018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.161662 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.161664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.162331 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.162334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.162752 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.162752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.162821 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.163908 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.163914 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.164445 (+ 531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.166216 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.166217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.166218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.166402 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.166842 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.166843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.166843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.166904 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.167129 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.167130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.167170 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.167234 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.167471 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":90,"thread_start_us":537,"threads_started":1}
I20260811 02:37:43.169454 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.159951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.2.251.62:41783)
0811 02:37:43.160060 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.160062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.160499 (+ 437us) server_negotiation.cc:407] Connection header received
0811 02:37:43.160539 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.160540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.160554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.160629 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.161083 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.162452 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.163425 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.163632 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.163690 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.164684 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.164692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.164693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.164710 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.167450 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.168285 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.168286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.168287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.168380 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.168826 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.168828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.168828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.169001 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.169074 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.169234 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.169344 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 02:37:43.169471 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.159802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:41764)
0811 02:37:43.160373 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.160381 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.160447 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.160637 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.160638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.160876 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.160994 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.160997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.162633 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.162634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.163322 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.163325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.163697 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.163698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.163767 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.164144 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.164149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.164532 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.167856 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.167858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.167859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.168139 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.168642 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.168644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.168644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.168730 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.169076 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.169078 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.169131 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.169198 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.169345 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":513,"threads_started":1}
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.62@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
I20260811 02:37:43.181959 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.172554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.2.251.61:43011)
0811 02:37:43.172604 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.172606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.172749 (+ 143us) server_negotiation.cc:407] Connection header received
0811 02:37:43.172846 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.172847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.172880 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.172931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.173435 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.173982 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.174800 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.174990 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.175243 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.177955 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.177957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.177958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.177968 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.180144 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.180742 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.180743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.180744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.180859 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.181156 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.181157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.181158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.181345 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.181423 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.181665 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.181833 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:37:43.182150 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.172465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58708)
0811 02:37:43.172635 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.172673 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.172765 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.172943 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.172948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.173263 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.173346 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.173348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.174097 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.174098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.174695 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.174697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.175245 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.175247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.175352 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.175926 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.175932 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.177828 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.180333 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.180347 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.180351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.180630 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.180981 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.180983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.180983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.181066 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.181448 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.181458 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.181527 (+ 69us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.181618 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.182022 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260811 02:37:43.183264 13105 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.251.61:43011: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a7126980727f4079a8447531f08e8269"
member_type: NON_VOTER
last_known_addr {
host: "127.2.251.61"
port: 43011
}
attrs {
promote: true
}
}
dest_uuid: "c7c088fc74e3491fb0b81d75400b7332"
cas_config_opid_index: -1
I20260811 02:37:43.183694 13105 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } }
I20260811 02:37:43.184518 14052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a7126980727f4079a8447531f08e8269 (127.2.251.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } }
I20260811 02:37:43.184713 13958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:43.185791 13958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Refusing update from remote peer c7c088fc74e3491fb0b81d75400b7332: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 02:37:43.185859 14052 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.251.61:43011
I20260811 02:37:43.186105 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } }
I20260811 02:37:43.186194 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:43.186232 13889 master.cc:537] Master@127.2.251.61:43011 shutting down...
I20260811 02:37:43.186297 14052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 02:37:43.187479 13083 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260811 02:37:43.188355 13083 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 -> Peer a7126980727f4079a8447531f08e8269 (127.2.251.61:43011): Couldn't send request to peer a7126980727f4079a8447531f08e8269. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 02:37:43.189934 13889 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:43.190050 13889 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:43.190110 13889 tablet_replica.cc:330] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: stopping tablet replica
W20260811 02:37:43.191812 13018 connection.cc:535] client connection to 127.2.251.61:43011 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 02:37:43.193005 13889 master.cc:552] Master@127.2.251.61:43011 shutdown complete.
I20260811 02:37:43.193061 13889 master_runner.cc:305] Clearing existing system tablet
I20260811 02:37:43.193202 13889 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.193252 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.194038 13889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:37:43.194612 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.194839 13889 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.194908 13889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:43.194933 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
uuid: "a7126980727f4079a8447531f08e8269"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "d01c546eaed4804ed279ffb99eb14f2a"
server_key_iv: "4690c17d4fbbe910f5656b04305addfc"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.195930 13889 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:43.197646 13889 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 02:37:43.197777 13889 log.cc:1196] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 02:37:43.198010 13889 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Deleting consensus metadata
I20260811 02:37:43.198171 13889 master_runner.cc:315] Copying system tablet from 127.2.251.62:41783
I20260811 02:37:43.198899 13889 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: tablet copy: Beginning tablet copy session from remote peer at address 127.2.251.62:41783
I20260811 02:37:43.206692 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.199538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.2.251.62:41783)
0811 02:37:43.199637 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.199639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.199646 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:37:43.199671 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.199672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.199683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.199730 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.200080 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.200627 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.201500 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.201810 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.201878 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.203908 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.203910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.203910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.203918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.205519 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.205929 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.205930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.205930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.205990 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.206155 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.206156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.206156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.206280 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.206339 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.206496 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.206584 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 02:37:43.206717 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.199347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:41766)
0811 02:37:43.199498 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.199509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.199565 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.199741 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.199742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.199952 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.200001 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.200003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.200712 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.200713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.201423 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.201426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.202123 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.202126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.202231 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.203413 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.203418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.203814 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.205605 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.205607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.205608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.205849 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.206042 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.206043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.206043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.206104 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.206335 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.206336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.206395 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.206464 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.206586 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260811 02:37:43.207325 13135 tablet_copy_service.cc:140] P c7c088fc74e3491fb0b81d75400b7332: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a7126980727f4079a8447531f08e8269 ({username='oryx', principal='oryx/127.2.251.61@KRBTEST.COM'} at 127.0.0.1:41766)
I20260811 02:37:43.207464 13135 tablet_copy_service.cc:161] P c7c088fc74e3491fb0b81d75400b7332: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a7126980727f4079a8447531f08e8269 at {username='oryx', principal='oryx/127.2.251.61@KRBTEST.COM'} at 127.0.0.1:41766: session id = a7126980727f4079a8447531f08e8269-00000000000000000000000000000000
I20260811 02:37:43.208357 13135 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 02:37:43.209328 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 02:37:43.210989 13889 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: tablet copy: Starting download of 0 data blocks...
I20260811 02:37:43.211167 13889 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: tablet copy: Starting download of 1 WAL segments...
I20260811 02:37:43.213042 13889 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 02:37:43.214448 13135 tablet_copy_service.cc:342] P c7c088fc74e3491fb0b81d75400b7332: Request end of tablet copy session a7126980727f4079a8447531f08e8269-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.251.61@KRBTEST.COM'} at 127.0.0.1:41766
I20260811 02:37:43.214532 13135 tablet_copy_service.cc:433] P c7c088fc74e3491fb0b81d75400b7332: ending tablet copy session a7126980727f4079a8447531f08e8269-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a7126980727f4079a8447531f08e8269
I20260811 02:37:43.218357 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:43.219743 14075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.219995 14074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.220161 13889 server_base.cc:1034] running on GCE node
W20260811 02:37:43.220028 14079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.220331 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:43.220484 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:43.221603 13889 hybrid_clock.cc:648] HybridClock initialized: now 1786415863221596 us; error 29 us; skew 500 ppm
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:43.223205 13889 init.cc:377] Logged in from keytab as oryx/127.2.251.61@KRBTEST.COM (short username oryx)
I20260811 02:37:43.223505 13889 webserver.cc:460] Webserver started at http://127.2.251.61:33617/ using document root <none> and password file <none>
I20260811 02:37:43.223623 13889 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.223656 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.224879 13889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:43.225562 14089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.225678 13889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.225760 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
uuid: "a7126980727f4079a8447531f08e8269"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "d01c546eaed4804ed279ffb99eb14f2a"
server_key_iv: "4690c17d4fbbe910f5656b04305addfc"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.225816 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.250264 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:43.252096 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:43.257542 13889 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.61:43011
I20260811 02:37:43.257747 14176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.61:43011 every 8 connection(s)
I20260811 02:37:43.258008 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
I20260811 02:37:43.259172 14182 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:37:43.259769 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Bootstrap starting.
I20260811 02:37:43.263191 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:37:43.263454 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: Bootstrap complete.
I20260811 02:37:43.263710 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } }
I20260811 02:37:43.263797 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7126980727f4079a8447531f08e8269, State: Initialized, Role: LEARNER
I20260811 02:37:43.263825 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } }
I20260811 02:37:43.264008 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:43.264081 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } }
I20260811 02:37:43.264546 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: LEARNER
I20260811 02:37:43.265508 14203 catalog_manager.cc:1259] Loaded cluster ID: fa29a9b4fca3464db553767f6a8aa3e7
I20260811 02:37:43.266548 14203 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: loading cluster ID for follower catalog manager: success
I20260811 02:37:43.268081 14203 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: acquiring CA information for follower catalog manager: success
I20260811 02:37:43.268501 14203 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:37:43.303754 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.293685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.2.251.61:43011)
0811 02:37:43.293882 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.293884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.293906 (+ 22us) server_negotiation.cc:407] Connection header received
0811 02:37:43.293940 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.293941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.293955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.294033 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.294640 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.295669 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.296552 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.296929 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.297013 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.299291 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.299293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.299293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.299304 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.302407 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.302904 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.302906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.302906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.302981 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.303161 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.303162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.303162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.303294 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.303389 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.303516 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.303613 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.2.251.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.60.keytab.
Entry for principal oryx/127.2.251.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.60.keytab.
I20260811 02:37:43.339614 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:37315
--webserver_interface=127.2.251.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.60.keytab
--principal=oryx/127.2.251.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:43.437161 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:43.437705 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:43.437883 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:43.439525 14209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:43.439586 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:43.439606 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:43.439622 14209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:43.439639 14209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:43.441588 14209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.60.keytab
--principal=oryx/127.2.251.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.251.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:43.441944 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:43.442220 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:43.447754 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.447754 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.447865 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.448179 14209 server_base.cc:1034] running on GCE node
I20260811 02:37:43.448683 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:43.449072 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:43.450279 14209 hybrid_clock.cc:648] HybridClock initialized: now 1786415863450265 us; error 38 us; skew 500 ppm
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:43.453922 14209 init.cc:377] Logged in from keytab as oryx/127.2.251.60@KRBTEST.COM (short username oryx)
I20260811 02:37:43.454697 14209 webserver.cc:460] Webserver started at http://127.2.251.60:35619/ using document root <none> and password file <none>
I20260811 02:37:43.454994 14209 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.455204 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.455442 14209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:43.456751 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/instance:
uuid: "50baeab390ed42f3be30cfbae6840004"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "604ccdf64a910b184219e12dc21f19ef"
server_key_iv: "56afdc9209260e62b11611c90d83c2cd"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.457149 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal/instance:
uuid: "50baeab390ed42f3be30cfbae6840004"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "604ccdf64a910b184219e12dc21f19ef"
server_key_iv: "56afdc9209260e62b11611c90d83c2cd"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.458837 14209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260811 02:37:43.460166 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.460453 14209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:43.460569 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
uuid: "50baeab390ed42f3be30cfbae6840004"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "604ccdf64a910b184219e12dc21f19ef"
server_key_iv: "56afdc9209260e62b11611c90d83c2cd"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.460654 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.480963 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:43.484997 14209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:43.485250 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:43.491024 14209 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.60:37315
I20260811 02:37:43.491024 14313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.60:37315 every 8 connection(s)
I20260811 02:37:43.491541 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
I20260811 02:37:43.492714 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:43.494643 14315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:37:43.497937 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 14209
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.60@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:43.507246 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.495775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.2.251.62:41783)
0811 02:37:43.495894 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.495896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.496084 (+ 188us) server_negotiation.cc:407] Connection header received
0811 02:37:43.496711 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.496713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.496729 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.496794 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.497917 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.498795 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.500487 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.500857 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.500955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.503879 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.503882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.503882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.503892 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.505489 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.506097 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.506099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.506099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.506168 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.506479 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.506480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.506480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.506630 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.506689 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.507032 (+ 343us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.507139 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 02:37:43.507282 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.495645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:41774)
0811 02:37:43.495905 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.495946 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.496577 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.496891 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.496895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.497437 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.497786 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.497794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.498950 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.498951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.500303 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.500307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.501013 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.501016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.501352 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.501997 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.502007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.503760 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.505613 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.505621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.505624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.505977 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.506248 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.506249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.506249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.506391 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.506698 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.506706 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.506872 (+ 166us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.506964 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.507100 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":120,"threads_started":1}
I20260811 02:37:43.508940 14315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.251.60@KRBTEST.COM
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.60@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
I20260811 02:37:43.513831 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.501200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.2.251.60:37315)
0811 02:37:43.502766 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.502771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.502799 (+ 28us) server_negotiation.cc:407] Connection header received
0811 02:37:43.502833 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.502836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.502860 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.502953 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.503619 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.504608 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.505748 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.506215 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.506289 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.509883 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.509893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.509894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.509902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.511622 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.512152 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.512156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.512156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.512374 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.512792 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.512794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.512795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.513163 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.513450 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.513524 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.513663 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":699,"threads_started":1}
I20260811 02:37:43.517441 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.509291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58750)
0811 02:37:43.509347 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.509357 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.509394 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.509573 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.509573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.509750 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.509819 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.509821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.510496 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.510497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.511095 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.511098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.511459 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.511466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.511542 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.511961 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.511967 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.513898 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.516137 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.516139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.516139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.516342 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.516591 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.516591 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.516592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.516659 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.517078 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.517079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.517138 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.517208 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.517339 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260811 02:37:43.517482 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.509392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.2.251.61:43011)
0811 02:37:43.509462 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.509465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.509472 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:37:43.509494 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.509495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.509510 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.509556 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.509890 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.510390 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.511180 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.511388 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.511453 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.514028 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.514031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.514031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.514043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.516008 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.516440 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.516441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.516442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.516519 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.516782 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.516784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.516784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.516944 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.517038 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.517246 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.517401 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:37:43.518239 14315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.60@KRBTEST.COM for oryx/127.2.251.60@KRBTEST.COM
I20260811 02:37:43.527026 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.518638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:47944)
0811 02:37:43.518746 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.518760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.518817 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.518929 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.518930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.519084 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.519135 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.519138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.519803 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.519804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.520341 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.520342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.520850 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.520851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.520917 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.521389 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.521394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.522934 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.525167 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.525169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.525170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.525484 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.526130 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.526131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.526132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.526234 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.526655 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.526656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.526700 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.526760 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.526902 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 02:37:43.527439 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.518757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.2.251.60:37315)
0811 02:37:43.518816 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.518818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.518826 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:43.518867 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.518868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.518879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.518924 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.519183 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.519695 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.520432 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.520728 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.520814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.523060 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.523062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.523063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.523072 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.525016 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.525915 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.525916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.525917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.525994 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.526329 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.526331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.526331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.526560 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.526645 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.526793 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.526969 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":41}
I20260811 02:37:43.529980 14315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Bootstrap starting.
I20260811 02:37:43.531203 14315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:43.531653 14315 log.cc:824] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:43.532862 14315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: No bootstrap required, opened a new log
I20260811 02:37:43.534667 14315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } }
I20260811 02:37:43.534838 14315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:43.534873 14315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50baeab390ed42f3be30cfbae6840004, State: Initialized, Role: FOLLOWER
I20260811 02:37:43.535034 14315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } }
I20260811 02:37:43.535369 14323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } } }
I20260811 02:37:43.535446 14315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:43.535499 14323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 02:37:43.538017 14341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:37:43.538094 14341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:37:43.541812 14209 master_runner.cc:428] Detected that this master 50baeab390ed42f3be30cfbae6840004 is joining an existing cluster
I20260811 02:37:43.541908 14209 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.251.60:37315
I20260811 02:37:43.548345 13408 heartbeater.cc:503] Master 127.2.251.62:41783 was elected leader, sending a full tablet report...
I20260811 02:37:43.565801 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 13206
I20260811 02:37:43.592139 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.544277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.2.251.60:37315)
0811 02:37:43.544327 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.544329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.544416 (+ 87us) server_negotiation.cc:407] Connection header received
0811 02:37:43.544492 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.544493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.544502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.544528 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.545414 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.546321 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.547671 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.547944 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.547994 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.588265 (+ 40271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.588267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.588267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.588275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.590948 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591364 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.591365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.591366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.591416 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591584 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.591585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.591585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.591723 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.591764 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.591944 (+ 180us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.592033 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260811 02:37:43.592465 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.544117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:47952)
0811 02:37:43.544368 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.544375 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.544433 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.544618 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.544618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.545209 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.545340 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.545341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.546471 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.546472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.547431 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.547435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.548104 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.548105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.548172 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.548745 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.548751 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.587455 (+ 38704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.591066 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.591068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.591269 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.591475 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.591475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.591524 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.591802 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.591802 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.591842 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.592236 (+ 394us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.592370 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":37,"thread_start_us":77,"threads_started":1}
I20260811 02:37:43.593266 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.543926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.2.251.61:43011)
0811 02:37:43.543995 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.543998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.544358 (+ 360us) server_negotiation.cc:407] Connection header received
0811 02:37:43.544500 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.544501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.544514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.544545 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.545450 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.546324 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.548006 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.548299 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.548346 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.589555 (+ 41209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.589558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.589559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.589573 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.591437 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591807 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.591808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.591809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.591868 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.592058 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.592058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.592059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.592187 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.592230 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.593090 (+ 860us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.593178 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:37:43.593259 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.543789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58764)
0811 02:37:43.544271 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.544284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.544412 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.544616 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.544617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.545204 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.545321 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.545324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.546471 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.546472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.547867 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.547869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.548894 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.548894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.549191 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.549585 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.549588 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.588178 (+ 38590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.591525 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.591526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.591706 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.591939 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.591939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.591940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.591990 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.592917 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.592918 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.592983 (+ 65us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.593039 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.593154 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":119,"thread_start_us":159,"threads_started":1}
I20260811 02:37:43.596707 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.543873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.2.251.62:41783)
0811 02:37:43.543939 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.543942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.544297 (+ 355us) server_negotiation.cc:407] Connection header received
0811 02:37:43.544432 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.544434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.544450 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.544485 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.545479 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.546339 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.547793 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.548141 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.548190 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.593499 (+ 45309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.593502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.593502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.593514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.595437 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.595850 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.595851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.595852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.595933 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.596137 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.596138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.596138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.596289 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.596368 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.596513 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.596605 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 02:37:43.596720 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.543783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:41778)
0811 02:37:43.544178 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.544192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.544338 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.544587 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.544588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.545229 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.545335 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.545339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.546464 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.546466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.547666 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.547670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.548650 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.548651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.548738 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.549161 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.549166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.587868 (+ 38702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.595548 (+ 7680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.595550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.595550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.595768 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.596009 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.596010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.596010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.596076 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.596352 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.596353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.596403 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.596479 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.596604 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":78,"thread_start_us":289,"threads_started":1}
I20260811 02:37:43.600194 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:34981
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:41783
--webserver_port=44453
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415862, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.62@KRBTEST.COM for oryx/127.2.251.60@KRBTEST.COM
I20260811 02:37:43.609041 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.599473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:47954)
0811 02:37:43.599572 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.599583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.599622 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.599829 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.599830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.600026 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.600087 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.600089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.600939 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.600941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.601889 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.601893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.602458 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.602459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.602532 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.602997 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.603003 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.604708 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.607407 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.607410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.607411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.607805 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.608135 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.608136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.608137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.608234 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.608598 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.608599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.608688 (+ 89us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.608785 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.608922 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260811 02:37:43.609570 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.599634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.2.251.60:37315)
0811 02:37:43.599698 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.599700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.599708 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:43.599747 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.599748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.599759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.599812 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.600186 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.600828 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.602013 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.602327 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.602392 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.604877 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.604881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.604881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.604896 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.607254 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.607933 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.607935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.607935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.608036 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.608330 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.608330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.608331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.608517 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.608594 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.608834 (+ 240us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.608944 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:37:43.610170 13105 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.251.60:37315: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50baeab390ed42f3be30cfbae6840004"
member_type: NON_VOTER
last_known_addr {
host: "127.2.251.60"
port: 37315
}
attrs {
promote: true
}
}
dest_uuid: "c7c088fc74e3491fb0b81d75400b7332"
cas_config_opid_index: 5
I20260811 02:37:43.610519 13105 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } }
I20260811 02:37:43.611343 14052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 50baeab390ed42f3be30cfbae6840004 (127.2.251.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.611507 14273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:43.612807 14052 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.251.60:37315
I20260811 02:37:43.612916 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.613149 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:43.613605 14273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Refusing update from remote peer c7c088fc74e3491fb0b81d75400b7332: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 02:37:43.614210 14209 master.cc:537] Master@127.2.251.60:37315 shutting down...
I20260811 02:37:43.615154 14053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 02:37:43.616210 13083 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 -> Peer 50baeab390ed42f3be30cfbae6840004 (127.2.251.60:37315): Couldn't send request to peer 50baeab390ed42f3be30cfbae6840004. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 02:37:43.619959 14209 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:43.620111 14209 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:43.620158 14209 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: stopping tablet replica
I20260811 02:37:43.624775 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.611130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58780)
0811 02:37:43.616825 (+ 5695us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.616846 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.616965 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.617248 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.617250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.617253 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:43.617316 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.617319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.618304 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.618306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.620364 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.620373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.623428 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.623431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.624190 (+ 759us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.624256 (+ 66us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.624339 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.624538 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5662}
I20260811 02:37:43.625527 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.611282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.2.251.61:43011)
0811 02:37:43.614842 (+ 3560us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.614845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.616948 (+ 2103us) server_negotiation.cc:407] Connection header received
0811 02:37:43.617040 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.617041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.617073 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.617154 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:43.617392 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.618185 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.620564 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.621966 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.623196 (+ 1230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.623345 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.625229 (+ 1884us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.625315 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3519}
I20260811 02:37:43.625805 14209 master.cc:552] Master@127.2.251.60:37315 shutdown complete.
I20260811 02:37:43.625885 14209 master_runner.cc:305] Clearing existing system tablet
I20260811 02:37:43.626080 14209 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.626140 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.626428 14137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 LEARNER]: Refusing update from remote peer c7c088fc74e3491fb0b81d75400b7332: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 02:37:43.626906 14053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 02:37:43.627156 14209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:43.627873 14137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 50baeab390ed42f3be30cfbae6840004 (127.2.251.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.630008 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7c088fc74e3491fb0b81d75400b7332. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.630152 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: LEARNER
I20260811 02:37:43.630204 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: NON_VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: true } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.630257 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: LEARNER
I20260811 02:37:43.630539 14053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: attempting to promote NON_VOTER a7126980727f4079a8447531f08e8269 to VOTER
I20260811 02:37:43.630533 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.630764 14209 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.630815 14209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260811 02:37:43.630837 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
uuid: "50baeab390ed42f3be30cfbae6840004"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "604ccdf64a910b184219e12dc21f19ef"
server_key_iv: "56afdc9209260e62b11611c90d83c2cd"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.630892 14053 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } }
I20260811 02:37:43.632025 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.631697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:47960)
0811 02:37:43.631903 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.631981 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260811 02:37:43.632038 14209 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:43.632623 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.632280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:47974)
0811 02:37:43.632505 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.632573 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260811 02:37:43.632714 13083 consensus_peers.cc:487] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 -> Peer 50baeab390ed42f3be30cfbae6840004 (127.2.251.60:37315): Couldn't send request to peer 50baeab390ed42f3be30cfbae6840004. Status: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 02:37:43.632912 14137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 LEARNER]: Refusing update from remote peer c7c088fc74e3491fb0b81d75400b7332: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 02:37:43.633204 14052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 02:37:43.634001 14209 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 02:37:43.634086 14209 log.cc:1196] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 02:37:43.634136 14052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a7126980727f4079a8447531f08e8269 (127.2.251.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.634321 14209 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Deleting consensus metadata
I20260811 02:37:43.634482 14209 master_runner.cc:315] Copying system tablet from 127.2.251.62:41783
I20260811 02:37:43.635192 14209 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: tablet copy: Beginning tablet copy session from remote peer at address 127.2.251.62:41783
I20260811 02:37:43.635273 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.635354 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:43.635558 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.635623 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:43.636054 14137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a7126980727f4079a8447531f08e8269 (127.2.251.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.637275 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.637390 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:37:43.647269 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.635713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.2.251.62:41783)
0811 02:37:43.635933 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.635936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.637892 (+ 1956us) server_negotiation.cc:407] Connection header received
0811 02:37:43.638019 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.638020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.638033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.638062 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.638553 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.639505 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.640925 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.641268 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.641328 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.643059 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.643062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.643062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.643073 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.645231 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.646159 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.646161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.646161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.646245 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.646522 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.646523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.646523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.646687 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.646760 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.647041 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.647145 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260811 02:37:43.647338 14376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.635805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:41786)
0811 02:37:43.637812 (+ 2007us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.637827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.637937 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.638134 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.638135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.638361 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.638433 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.638436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.639868 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.639870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.640764 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.640769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.641452 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.641453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.641538 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.642354 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.642362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.642875 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.645374 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.645376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.645376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.645970 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.646345 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.646346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.646346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.646437 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.646789 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.646791 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.646889 (+ 98us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.646980 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.647181 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1978,"thread_start_us":81,"threads_started":1}
I20260811 02:37:43.648052 13135 tablet_copy_service.cc:140] P c7c088fc74e3491fb0b81d75400b7332: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50baeab390ed42f3be30cfbae6840004 ({username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:41786)
I20260811 02:37:43.648159 13135 tablet_copy_service.cc:161] P c7c088fc74e3491fb0b81d75400b7332: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50baeab390ed42f3be30cfbae6840004 at {username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:41786: session id = 50baeab390ed42f3be30cfbae6840004-00000000000000000000000000000000
I20260811 02:37:43.649149 13135 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 02:37:43.650107 14209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 02:37:43.651820 14209 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: tablet copy: Starting download of 0 data blocks...
I20260811 02:37:43.652002 14209 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: tablet copy: Starting download of 1 WAL segments...
I20260811 02:37:43.653020 14209 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 02:37:43.654386 13135 tablet_copy_service.cc:342] P c7c088fc74e3491fb0b81d75400b7332: Request end of tablet copy session 50baeab390ed42f3be30cfbae6840004-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:41786
I20260811 02:37:43.654475 13135 tablet_copy_service.cc:433] P c7c088fc74e3491fb0b81d75400b7332: ending tablet copy session 50baeab390ed42f3be30cfbae6840004-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50baeab390ed42f3be30cfbae6840004
I20260811 02:37:43.657976 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:43.659660 14379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.662750 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.663096 14388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.663379 14209 server_base.cc:1034] running on GCE node
I20260811 02:37:43.663508 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:43.663668 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:43.664815 14209 hybrid_clock.cc:648] HybridClock initialized: now 1786415863664804 us; error 25 us; skew 500 ppm
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:43.666697 14209 init.cc:377] Logged in from keytab as oryx/127.2.251.60@KRBTEST.COM (short username oryx)
I20260811 02:37:43.667002 14209 webserver.cc:460] Webserver started at http://127.2.251.60:43035/ using document root <none> and password file <none>
I20260811 02:37:43.667132 14209 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.667174 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.668011 14209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:43.668653 14402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.668836 14209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:43.668891 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
uuid: "50baeab390ed42f3be30cfbae6840004"
format_stamp: "Formatted at 2026-08-11 02:37:43 on dist-test-slave-cvwc"
server_key: "604ccdf64a910b184219e12dc21f19ef"
server_key_iv: "56afdc9209260e62b11611c90d83c2cd"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.668959 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.689222 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:43.691531 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:43.695319 14209 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.60:37315
I20260811 02:37:43.695371 14489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.60:37315 every 8 connection(s)
I20260811 02:37:43.695712 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
I20260811 02:37:43.696781 14490 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:37:43.697536 14490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Bootstrap starting.
I20260811 02:37:43.702314 14490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:37:43.702885 14490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Bootstrap complete.
I20260811 02:37:43.703207 14490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } }
I20260811 02:37:43.703322 14490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50baeab390ed42f3be30cfbae6840004, State: Initialized, Role: LEARNER
I20260811 02:37:43.703362 14490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } }
I20260811 02:37:43.703926 14490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:43.704145 14497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:43.704355 14497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: This master's current role is: LEARNER
I20260811 02:37:43.707063 14521 catalog_manager.cc:1259] Loaded cluster ID: fa29a9b4fca3464db553767f6a8aa3e7
I20260811 02:37:43.707180 14521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: loading cluster ID for follower catalog manager: success
I20260811 02:37:43.708410 14521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: acquiring CA information for follower catalog manager: success
I20260811 02:37:43.708776 14521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 02:37:43.722987 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:43.723232 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:43.723268 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:43.724593 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:43.724653 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:43.724694 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:43.726641 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:34981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=44453
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:43.726948 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:43.727206 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:43.733995 14528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.733961 14530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.734177 14533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.734534 14355 server_base.cc:1034] running on GCE node
I20260811 02:37:43.734779 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:43.735137 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:43.736321 14355 hybrid_clock.cc:648] HybridClock initialized: now 1786415863736289 us; error 72 us; skew 500 ppm
I20260811 02:37:43.739356 13627 heartbeater.cc:503] Master 127.2.251.62:41783 was elected leader, sending a full tablet report...
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:43.740085 14355 init.cc:377] Logged in from keytab as oryx/127.2.251.1@KRBTEST.COM (short username oryx)
I20260811 02:37:43.740724 14355 webserver.cc:460] Webserver started at http://127.2.251.1:44453/ using document root <none> and password file <none>
I20260811 02:37:43.740978 14355 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.741055 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.742799 14355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:43.743757 14542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.744122 14355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:43.744238 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "0e29543d60f5459bb151b5cfcb4e64dc"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "3bf723bdec5be672ac02d0d55dd4e93f"
server_key_iv: "3bbdbdf59fe0f3d73c8550b9ed223fc2"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.744664 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.785408 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:43.788964 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:43.789335 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:43.789669 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:43.790288 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:43.790347 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.790383 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:43.790400 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.797619 14355 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:34981
I20260811 02:37:43.797689 14741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:34981 every 8 connection(s)
I20260811 02:37:43.798127 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:43.799537 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 14355
I20260811 02:37:43.799721 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 13414
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.60@KRBTEST.COM
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:43.820143 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.798969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.2.251.1:36153)
0811 02:37:43.799169 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.799216 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.799916 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.804713 (+ 4797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.804713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.804849 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.804894 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.804895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.805993 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.805995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.807061 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.807064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.809530 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.809530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.809697 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.811226 (+ 1529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.811237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.814430 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.818127 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.818143 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.818147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.818608 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.818831 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.818831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.818832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.818977 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.819409 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.819417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.819598 (+ 181us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.819692 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.819872 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":116,"threads_started":1}
I20260811 02:37:43.820632 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.798975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:36153 (local address 127.2.251.60:37315)
0811 02:37:43.799141 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.799143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.799319 (+ 176us) server_negotiation.cc:407] Connection header received
0811 02:37:43.803037 (+ 3718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.803039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.803059 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.803146 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.804981 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.805876 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.807154 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.807398 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.807454 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.814611 (+ 7157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.814613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.814613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.814631 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.818028 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.818703 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.818704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.818704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.818775 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.819029 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.819030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.819030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.819167 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.819210 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.820422 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.820537 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260811 02:37:43.821072 14742 heartbeater.cc:347] Connected to a master server at 127.2.251.60:37315
I20260811 02:37:43.821185 14742 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:43.821393 14742 heartbeater.cc:511] Master 127.2.251.60:37315 requested a full tablet report, sending...
I20260811 02:37:43.821970 14434 ts_manager.cc:194] Registered new tserver with Master: 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981)
I20260811 02:37:43.822718 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.798969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:53843)
0811 02:37:43.799387 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.799399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.803192 (+ 3793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.803495 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.803500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.804278 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.804604 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.804610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.808278 (+ 3668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.808279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.809400 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.809404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.810426 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.810427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.810492 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.812775 (+ 2283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.812781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.815076 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.819234 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.819234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.819235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.820878 (+ 1643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.821129 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.821130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.821131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.821197 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.821470 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.821470 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.822415 (+ 945us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.822475 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.822598 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":876,"spinlock_wait_cycles":768,"thread_start_us":111,"threads_started":1}
I20260811 02:37:43.822738 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.798954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:53843 (local address 127.2.251.62:41783)
0811 02:37:43.799239 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.799241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.799477 (+ 236us) server_negotiation.cc:407] Connection header received
0811 02:37:43.803273 (+ 3796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.803275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.803287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.803311 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.807546 (+ 4235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.808187 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.810054 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.810311 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.810348 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.815209 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.815211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.815211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.815221 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.819084 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.820966 (+ 1882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.820968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.820968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.821055 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.821261 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.821262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.821262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.821391 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.821438 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.822534 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.822640 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260811 02:37:43.823065 14744 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41783
I20260811 02:37:43.823113 14744 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:43.823179 14744 heartbeater.cc:511] Master 127.2.251.62:41783 requested a full tablet report, sending...
I20260811 02:37:43.823427 13105 ts_manager.cc:194] Re-registered known tserver with Master: 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981)
I20260811 02:37:43.823805 13105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:53843
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
I20260811 02:37:43.828238 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:43099
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:41783
--webserver_port=38511
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:37:43.829931 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.816606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:59039 (local address 127.2.251.61:43011)
0811 02:37:43.816956 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:43.816958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:43.816971 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:43.817011 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:43.817012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:43.817028 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:43.817111 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:43.819839 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.820563 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.822363 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.822668 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.822744 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.826750 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:43.826752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:43.826753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:43.826763 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:43.828572 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.829057 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.829058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.829059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.829126 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:43.829305 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:43.829306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:43.829306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:43.829444 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:43.829510 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:43.829678 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 02:37:43.829836 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260811 02:37:43.829946 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:43.799306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.2.251.1:59039)
0811 02:37:43.799481 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:43.799490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:43.800492 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:43.819240 (+ 18748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:43.819243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:43.819596 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:43.819698 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.819701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.820671 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.820672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.822261 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:43.822265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:43.822847 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:43.822848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:43.822923 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:43.823506 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:43.823512 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:43.825607 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:43.828736 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.828738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.828739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.828967 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.829187 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:43.829188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:43.829188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:43.829244 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:43.829532 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:43.829533 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:43.829579 (+ 46us) client_negotiation.cc:769] Sending connection context
0811 02:37:43.829630 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 02:37:43.829810 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
I20260811 02:37:43.831037 14745 heartbeater.cc:347] Connected to a master server at 127.2.251.61:43011
I20260811 02:37:43.831120 14745 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:43.831202 14745 heartbeater.cc:511] Master 127.2.251.61:43011 requested a full tablet report, sending...
I20260811 02:37:43.831676 14119 ts_manager.cc:194] Registered new tserver with Master: 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981)
W20260811 02:37:43.935211 14753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:43.935451 14753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:43.935480 14753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:43.936843 14753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:43.936900 14753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:43.936930 14753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:43.938838 14753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:43099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=38511
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:43.939087 13876 heartbeater.cc:503] Master 127.2.251.62:41783 was elected leader, sending a full tablet report...
I20260811 02:37:43.939184 14753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:43.939468 14753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:37:43.945459 14753 server_base.cc:1034] running on GCE node
W20260811 02:37:43.945499 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.945448 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:43.945473 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:43.946164 14753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:43.946467 14753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:43.947723 14753 hybrid_clock.cc:648] HybridClock initialized: now 1786415863947698 us; error 36 us; skew 500 ppm
Aug 11 02:37:43 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:43.951654 14753 init.cc:377] Logged in from keytab as oryx/127.2.251.2@KRBTEST.COM (short username oryx)
I20260811 02:37:43.952235 14753 webserver.cc:460] Webserver started at http://127.2.251.2:38511/ using document root <none> and password file <none>
I20260811 02:37:43.952497 14753 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:43.952562 14753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:43.954301 14753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:43.955343 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.955574 14753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:43.955655 14753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "e039f61b46234dc88ed53dc615dfb572"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "1e42cc45f36b940cac51953a3a683f44"
server_key_iv: "e709b6294a36a684fea4d457040e4560"
server_key_version: "encryptionkey@0"
I20260811 02:37:43.956072 14753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:43.989195 14753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:43.992293 14753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:43.992457 14753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:43.992745 14753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:43.993119 14753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:43.993162 14753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:43.993186 14753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:43.993197 14753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:44.000520 14753 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:43099
I20260811 02:37:44.000593 14968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:43099 every 8 connection(s)
I20260811 02:37:44.001261 14753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:44.006886 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 14753
I20260811 02:37:44.007117 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 13635
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.60@KRBTEST.COM
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:44.021332 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.002524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:35501 (local address 127.2.251.60:37315)
0811 02:37:44.002589 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.002592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.003154 (+ 562us) server_negotiation.cc:407] Connection header received
0811 02:37:44.004627 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.004629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.004650 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.004682 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.006031 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.006924 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.013413 (+ 6489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.013776 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.013824 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.017806 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.017809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.017810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.017821 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.019469 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.020068 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.020070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.020071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.020134 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.020485 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.020486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.020486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.020646 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.020692 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.021122 (+ 430us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.021229 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:37:44.022068 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.002531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.2.251.2:35501)
0811 02:37:44.002935 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.003008 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.004462 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.004819 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.004824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.005485 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:44.005865 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.005877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.007080 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.007081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.013183 (+ 6102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.013195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.013942 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.013943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.014117 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.014933 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:44.014948 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:44.017607 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:44.019566 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.019582 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.019586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.019965 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.020217 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.020218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.020219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.020395 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.020743 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:44.020751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:44.020958 (+ 207us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.021501 (+ 543us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.021765 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":30,"thread_start_us":152,"threads_started":1}
I20260811 02:37:44.022145 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.002524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:48327 (local address 127.2.251.62:41783)
0811 02:37:44.002589 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.002592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.003194 (+ 602us) server_negotiation.cc:407] Connection header received
0811 02:37:44.004629 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.004632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.004650 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.004710 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.006031 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.006900 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.015494 (+ 8594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.015882 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.015936 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.019293 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.019295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.019296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.019307 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.020923 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.021301 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.021302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.021302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.021360 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.021553 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.021554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.021554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.021690 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.021766 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.021969 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.022060 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:37:44.022503 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.002531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:48327)
0811 02:37:44.002937 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.003008 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.004462 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.004819 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.004824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.005411 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:44.005865 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.005876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.007080 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.007081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.015340 (+ 8259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.015345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.016279 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.016286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.016425 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.017085 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:44.017091 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:44.019143 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:44.021018 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.021019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.021019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.021217 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.021424 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.021425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.021425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.021484 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.021799 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:44.021800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:44.021844 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.022273 (+ 429us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.022394 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":61,"thread_start_us":115,"threads_started":1}
I20260811 02:37:44.022620 14969 heartbeater.cc:347] Connected to a master server at 127.2.251.60:37315
I20260811 02:37:44.022732 14969 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:44.022955 14969 heartbeater.cc:511] Master 127.2.251.60:37315 requested a full tablet report, sending...
I20260811 02:37:44.023460 14434 ts_manager.cc:194] Registered new tserver with Master: e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099)
I20260811 02:37:44.029150 14974 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41783
I20260811 02:37:44.029253 14974 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:44.029346 14974 heartbeater.cc:511] Master 127.2.251.62:41783 requested a full tablet report, sending...
I20260811 02:37:44.029779 13105 ts_manager.cc:194] Re-registered known tserver with Master: e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099)
I20260811 02:37:44.030265 13105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:48327
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415863, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
I20260811 02:37:44.036346 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.003102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:54263 (local address 127.2.251.61:43011)
0811 02:37:44.003158 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.003161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.003402 (+ 241us) server_negotiation.cc:407] Connection header received
0811 02:37:44.004649 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.004651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.004666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.004698 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.006026 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.007093 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.030501 (+ 23408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.030913 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.030972 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.033593 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.033594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.033595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.033604 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.035170 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.035559 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.035561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.035561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.035634 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.035810 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.035811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.035812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.035941 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.035998 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.036140 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.036232 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:37:44.036363 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.002963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.2.251.2:54263)
0811 02:37:44.003276 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.003289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.004482 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.004819 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.004824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.005458 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:44.005862 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.005875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.029212 (+ 23337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.029216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.030320 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.030327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.031088 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.031089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.031174 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.031735 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:44.031741 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:44.033471 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:44.035295 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.035297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.035298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.035489 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.035693 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.035694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.035694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.035758 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.035998 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:44.035999 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:44.036051 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.036111 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.036242 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":42,"thread_start_us":103,"threads_started":1}
I20260811 02:37:44.036923 14975 heartbeater.cc:347] Connected to a master server at 127.2.251.61:43011
I20260811 02:37:44.036980 14975 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:44.037051 14975 heartbeater.cc:511] Master 127.2.251.61:43011 requested a full tablet report, sending...
I20260811 02:37:44.037477 14119 ts_manager.cc:194] Registered new tserver with Master: e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099)
I20260811 02:37:44.037791 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:41133
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:41783
--webserver_port=41257
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:44.131145 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:44.131386 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:44.131412 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:44.132649 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:44.132689 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:44.132716 14981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:44.134558 14981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:41133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=41257
--tserver_master_addrs=127.2.251.62:41783,127.2.251.61:43011,127.2.251.60:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:44.134809 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:44.135047 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:44.140918 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:44.140941 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:44.141423 14981 server_base.cc:1034] running on GCE node
W20260811 02:37:44.140992 14989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:44.141992 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:44.142338 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:44.143536 14981 hybrid_clock.cc:648] HybridClock initialized: now 1786415864143522 us; error 33 us; skew 500 ppm
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415864, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:44.146499 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.143230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:47988)
0811 02:37:44.143374 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.143388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.143485 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.143727 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.143729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.143732 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:44.143795 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.143798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.144719 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.144720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.145513 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.145516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.146232 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.146233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.146304 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.146309 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.146366 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.146408 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":49,"threads_started":1}
I20260811 02:37:44.146503 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.143353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.2.251.60:37315)
0811 02:37:44.143405 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.143407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.143461 (+ 54us) server_negotiation.cc:407] Connection header received
0811 02:37:44.143604 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.143606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.143641 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.143717 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:44.143857 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.144381 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.145609 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.146122 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.146203 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.146278 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.146383 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.146429 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:44.146893 14981 init.cc:377] Logged in from keytab as oryx/127.2.251.3@KRBTEST.COM (short username oryx)
I20260811 02:37:44.146955 14447 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 02:37:44.147140 14497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7c088fc74e3491fb0b81d75400b7332. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:44.147256 14497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: This master's current role is: LEARNER
I20260811 02:37:44.147500 14981 webserver.cc:460] Webserver started at http://127.2.251.3:41257/ using document root <none> and password file <none>
I20260811 02:37:44.147789 14981 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:44.147850 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:44.149492 14981 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:44.150506 15009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:44.150743 14981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:44.150835 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "cfc87615c43047858efb37206c32fe1f"
format_stamp: "Formatted at 2026-08-11 02:37:42 on dist-test-slave-cvwc"
server_key: "6b6fd97d5fe1cc824c94ff222025a3d4"
server_key_iv: "903cd530b28b3475e46583ab8bd30ef6"
server_key_version: "encryptionkey@0"
I20260811 02:37:44.151343 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:44.167572 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:44.170887 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:44.171062 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:44.171298 14981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:44.171648 14981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:44.171685 14981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:44.171716 14981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:44.171733 14981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:44.178656 14981 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:41133
I20260811 02:37:44.178821 15209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:41133 every 8 connection(s)
I20260811 02:37:44.179200 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:44.185938 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 14981
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415864, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.61@KRBTEST.COM
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415864, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.60@KRBTEST.COM
Aug 11 02:37:44 dist-test-slave-cvwc krb5kdc[13021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415864, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:44.193914 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.180432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.2.251.3:58905)
0811 02:37:44.180566 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.180576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.181612 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.182011 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.182012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.182427 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:44.182638 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.182644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.185108 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.185109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.185666 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.185669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.186334 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.186335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.186428 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.186950 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:44.186955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:44.188998 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:44.191057 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.191070 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.191073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.191993 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.192570 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.192571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.192571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.192728 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.193039 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:44.193046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:44.193275 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.193413 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.193617 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":13,"thread_start_us":75,"threads_started":1}
I20260811 02:37:44.194397 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.180543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:58905 (local address 127.2.251.60:37315)
0811 02:37:44.180592 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.180594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.180627 (+ 33us) server_negotiation.cc:407] Connection header received
0811 02:37:44.181924 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.181925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.181942 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.181997 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.184234 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.185035 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.185794 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.186066 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.186131 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.189103 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.189104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.189104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.189112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.190968 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.192422 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.192424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.192424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.192491 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.192804 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.192806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.192806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.192943 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.193008 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.194205 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.194304 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:44.194761 15210 heartbeater.cc:347] Connected to a master server at 127.2.251.60:37315
I20260811 02:37:44.194866 15210 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:44.195080 15210 heartbeater.cc:511] Master 127.2.251.60:37315 requested a full tablet report, sending...
I20260811 02:37:44.195443 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.180605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.2.251.3:48499)
0811 02:37:44.180730 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.180739 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.181639 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.181944 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.181945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.182399 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:44.182637 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.182643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.183386 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.183387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.184768 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.184772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.185313 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.185313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.185368 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.186241 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:44.186260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:44.191280 (+ 5020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:44.193810 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.193811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.193811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.194039 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.194664 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.194665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.194665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.194744 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.195113 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:44.195114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:44.195162 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.195231 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.195342 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260811 02:37:44.195582 14434 ts_manager.cc:194] Registered new tserver with Master: cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133)
I20260811 02:37:44.195914 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.180091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:52503)
0811 02:37:44.180336 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.180381 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.181610 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.181921 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.181925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.182427 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:44.182638 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.182644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.184225 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.184226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.184780 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.184781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.185179 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.185180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.185329 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.189446 (+ 4117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:44.189451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:44.190928 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:44.194851 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.194852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.194852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.195047 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.195250 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:44.195251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:44.195251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:44.195305 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:44.195600 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:44.195601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:44.195642 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.195699 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.195797 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":7,"thread_start_us":103,"threads_started":1}
I20260811 02:37:44.196061 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.187661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.2.251.62:41783)
0811 02:37:44.187771 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.187773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.187782 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:44.187806 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.187807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.187818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.188503 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.188787 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.189612 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.190738 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.191066 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.191384 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.192625 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.192626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.192626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.192633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.194594 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.195009 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.195010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.195010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.195068 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.195270 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.195271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.195271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.195409 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.195835 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.195879 (+ 44us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.195968 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260811 02:37:44.196115 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.180819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:48499 (local address 127.2.251.61:43011)
0811 02:37:44.180889 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.180891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.180900 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:44.181828 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.181830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.181841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.181872 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.182751 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.183260 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.184882 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.185192 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.185232 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.191406 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.191408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.191408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.191418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.193269 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.194361 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.194362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.194363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.194427 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.194817 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.194817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.194818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.194962 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.195025 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.195943 (+ 918us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.196029 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 02:37:44.196184 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.180177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:52503 (local address 127.2.251.62:41783)
0811 02:37:44.180290 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.180292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.180495 (+ 203us) server_negotiation.cc:407] Connection header received
0811 02:37:44.181773 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.181775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.181791 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.181852 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.183377 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.184105 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.184886 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.185099 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.185168 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.191013 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.191015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.191015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.191025 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.194042 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.195125 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.195126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.195127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.195184 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.195384 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.195385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.195385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.195508 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.195572 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.196029 (+ 457us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.196112 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 02:37:44.196497 15212 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41783
I20260811 02:37:44.196555 15212 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:44.196624 15212 heartbeater.cc:511] Master 127.2.251.62:41783 requested a full tablet report, sending...
I20260811 02:37:44.196902 13105 ts_manager.cc:194] Re-registered known tserver with Master: cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133)
I20260811 02:37:44.197018 15214 heartbeater.cc:347] Connected to a master server at 127.2.251.61:43011
I20260811 02:37:44.197075 15214 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:44.197149 15214 heartbeater.cc:511] Master 127.2.251.61:43011 requested a full tablet report, sending...
I20260811 02:37:44.197347 13105 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:52503
I20260811 02:37:44.197444 14119 ts_manager.cc:194] Registered new tserver with Master: cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133)
I20260811 02:37:44.198985 13105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:44.201141 13105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:44.202163 14996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: NON_VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: true } } }
I20260811 02:37:44.202280 14996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:44.203254 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.188024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.2.251.60:37315)
0811 02:37:44.188149 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.188151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.188160 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:44.188183 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.188183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.188195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.188283 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.189014 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.190080 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.191531 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.191821 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.199100 (+ 7279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.199153 (+ 53us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.199153 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.199154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.199158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.201921 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.202328 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.202329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.202329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.202393 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.202613 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.202614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.202614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.202736 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.203030 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.203070 (+ 40us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.203158 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260811 02:37:44.203711 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.188001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.2.251.61:43011)
0811 02:37:44.192243 (+ 4242us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.192245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.192253 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:44.192277 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.192278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.192291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.193033 (+ 742us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.193081 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.193858 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.196389 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.196710 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.197309 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.198892 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.198893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.198893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.198897 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.201416 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.202167 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.202169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.202170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.202247 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.202474 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.202475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.202476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.202651 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.202755 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.203209 (+ 454us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.203588 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4216,"thread_start_us":62,"threads_started":1}
I20260811 02:37:44.204994 15248 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: attempting to promote NON_VOTER 50baeab390ed42f3be30cfbae6840004 to VOTER
I20260811 02:37:44.205205 15248 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:44.205739 14137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Refusing update from remote peer c7c088fc74e3491fb0b81d75400b7332: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 02:37:44.205948 14446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 LEARNER]: Refusing update from remote peer c7c088fc74e3491fb0b81d75400b7332: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 02:37:44.205916 15242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 02:37:44.206138 15242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 02:37:44.207050 15248 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 50baeab390ed42f3be30cfbae6840004 (127.2.251.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.207057 14137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 50baeab390ed42f3be30cfbae6840004 (127.2.251.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.208115 14996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.208160 15248 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.208204 14996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:44.208263 14996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.208310 14996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:44.208212 15248 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:44.208788 15255 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.208899 15255 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:37:44.208920 14446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 50baeab390ed42f3be30cfbae6840004 (127.2.251.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.210091 14497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7c088fc74e3491fb0b81d75400b7332" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:44.210318 14497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:37:44.215504 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.211550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.0.0.1:44520)
0811 02:37:44.211792 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.211802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.211851 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.212299 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.212299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.212302 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:44.212342 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.212343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.213521 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.213522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.214323 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.214326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.215244 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.215245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.215317 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.215323 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.215355 (+ 32us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.215405 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260811 02:37:44.215591 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.211423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:41133 (local address 127.0.0.1:41670)
0811 02:37:44.211527 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.211540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.211588 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.212161 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.212162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.212164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:44.212207 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.212209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.213330 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.213331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.214527 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.214530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.215410 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.215411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.215467 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.215471 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.215496 (+ 25us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.215542 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 02:37:44.215699 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.211562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.2.251.2:43099)
0811 02:37:44.211783 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.211789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.211801 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:44.211854 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.211856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.211872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.211949 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:44.212178 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.213401 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.214728 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.215370 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.215443 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.215534 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.215563 (+ 29us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.215604 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":130,"threads_started":1}
I20260811 02:37:44.215907 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.211962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.2.251.1:34981)
0811 02:37:44.212154 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.212158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.212168 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:44.212208 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.212210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.212226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.212304 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:44.212400 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.213458 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.214405 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.215128 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.215645 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.215740 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.215781 (+ 41us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.215828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260811 02:37:44.216125 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.211423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:43099 (local address 127.0.0.1:39648)
0811 02:37:44.211692 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.211703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.211784 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.212037 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.212038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.212040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:44.212100 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.212102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.213483 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.213484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.214653 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.214655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.215442 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.215443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.215505 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.215509 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.215549 (+ 40us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.216048 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":36,"thread_start_us":102,"threads_started":1}
I20260811 02:37:44.216168 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.211694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.2.251.3:41133)
0811 02:37:44.211979 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.211984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.211994 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:44.212036 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.212038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.212054 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.212160 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:44.212265 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.213258 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.214618 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.215322 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.215759 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.215842 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.215866 (+ 24us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.215914 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
I20260811 02:37:44.216742 14632 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7 (DEFAULT_TABLE table=test-table [id=bdc28f338c3a4ee9bfe80abc13d21c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:44.216830 15091 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7 (DEFAULT_TABLE table=test-table [id=bdc28f338c3a4ee9bfe80abc13d21c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:44.216830 14866 tablet_service.cc:1467] Processing CreateTablet for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7 (DEFAULT_TABLE table=test-table [id=bdc28f338c3a4ee9bfe80abc13d21c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:44.217157 14632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:44.217221 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:44.217208 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:44.220353 15264 tablet_bootstrap.cc:492] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Bootstrap starting.
I20260811 02:37:44.220963 15266 tablet_bootstrap.cc:492] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Bootstrap starting.
I20260811 02:37:44.221259 15264 tablet_bootstrap.cc:654] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:44.221678 15266 tablet_bootstrap.cc:654] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:44.221652 15264 log.cc:824] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:44.222047 15266 log.cc:824] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:44.222200 15270 tablet_bootstrap.cc:492] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Bootstrap starting.
I20260811 02:37:44.222965 15266 tablet_bootstrap.cc:492] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: No bootstrap required, opened a new log
I20260811 02:37:44.223042 15266 ts_tablet_manager.cc:1397] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 02:37:44.223531 15270 tablet_bootstrap.cc:654] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:44.224457 15270 log.cc:824] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:44.225088 15266 raft_consensus.cc:348] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.225327 15266 raft_consensus.cc:374] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:44.225368 15266 raft_consensus.cc:729] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e039f61b46234dc88ed53dc615dfb572, State: Initialized, Role: FOLLOWER
I20260811 02:37:44.225646 15264 tablet_bootstrap.cc:492] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: No bootstrap required, opened a new log
I20260811 02:37:44.225656 15270 tablet_bootstrap.cc:492] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: No bootstrap required, opened a new log
I20260811 02:37:44.225600 15266 consensus_queue.cc:260] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.225754 15264 ts_tablet_manager.cc:1397] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260811 02:37:44.225775 15270 ts_tablet_manager.cc:1397] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260811 02:37:44.226020 14974 heartbeater.cc:503] Master 127.2.251.62:41783 was elected leader, sending a full tablet report...
I20260811 02:37:44.226248 15266 ts_tablet_manager.cc:1428] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 02:37:44.228209 15270 raft_consensus.cc:348] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.228498 15270 raft_consensus.cc:374] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:44.228541 15270 raft_consensus.cc:729] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc87615c43047858efb37206c32fe1f, State: Initialized, Role: FOLLOWER
I20260811 02:37:44.228570 15264 raft_consensus.cc:348] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.228713 15270 consensus_queue.cc:260] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.228781 15264 raft_consensus.cc:374] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:44.228812 15264 raft_consensus.cc:729] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e29543d60f5459bb151b5cfcb4e64dc, State: Initialized, Role: FOLLOWER
I20260811 02:37:44.228952 15264 consensus_queue.cc:260] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.229137 15270 ts_tablet_manager.cc:1428] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:37:44.229171 15212 heartbeater.cc:503] Master 127.2.251.62:41783 was elected leader, sending a full tablet report...
I20260811 02:37:44.229274 15264 ts_tablet_manager.cc:1428] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260811 02:37:44.229404 14744 heartbeater.cc:503] Master 127.2.251.62:41783 was elected leader, sending a full tablet report...
W20260811 02:37:44.252698 14977 tablet.cc:2369] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:44.288703 15280 raft_consensus.cc:482] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:44.288851 15280 raft_consensus.cc:504] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.289238 15280 leader_election.cc:288] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133), e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099)
I20260811 02:37:44.292095 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.289488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:50405 (local address 127.2.251.2:43099)
0811 02:37:44.289533 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.289535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.289544 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:44.289621 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.289622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.289635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.289677 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:44.289874 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.290357 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.291351 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.291742 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.291806 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.291819 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.291973 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.292015 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:37:44.292099 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.289539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:60449 (local address 127.2.251.3:41133)
0811 02:37:44.289602 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.289604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.289612 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:44.289640 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.289641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.289653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.289706 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:44.289892 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.290353 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.291373 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.291751 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.291818 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.291837 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.291989 (+ 152us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.292031 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:37:44.292114 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.289418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:43099 (local address 127.2.251.1:50405)
0811 02:37:44.289491 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.289502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.289545 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.289683 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.289684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.289686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:44.289807 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.289809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.290440 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.290441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.291258 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.291260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.291824 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.291826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.291906 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.291910 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.291957 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.292013 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":29}
I20260811 02:37:44.292126 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.289432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:41133 (local address 127.2.251.1:60449)
0811 02:37:44.289491 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:44.289502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:44.289545 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:44.289745 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:44.289746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:44.289747 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:44.289808 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.289809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.290437 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.290438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.291270 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:44.291273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.291836 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.291837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:44.291910 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.291913 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:44.291966 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 02:37:44.292018 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":57}
I20260811 02:37:44.292618 14904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb5c5cbab64ed1a8c60dcfc0578bb7" candidate_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e039f61b46234dc88ed53dc615dfb572" is_pre_election: true
I20260811 02:37:44.292706 15123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb5c5cbab64ed1a8c60dcfc0578bb7" candidate_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc87615c43047858efb37206c32fe1f" is_pre_election: true
I20260811 02:37:44.292824 14904 raft_consensus.cc:2454] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e29543d60f5459bb151b5cfcb4e64dc in term 0.
I20260811 02:37:44.292861 15123 raft_consensus.cc:2454] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e29543d60f5459bb151b5cfcb4e64dc in term 0.
I20260811 02:37:44.293134 14551 leader_election.cc:302] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e29543d60f5459bb151b5cfcb4e64dc, e039f61b46234dc88ed53dc615dfb572; no voters:
I20260811 02:37:44.293288 15280 raft_consensus.cc:2781] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:44.293337 15280 raft_consensus.cc:482] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:44.293365 15280 raft_consensus.cc:3037] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:44.294175 15280 raft_consensus.cc:504] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.294333 15280 leader_election.cc:288] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [CANDIDATE]: Term 1 election: Requested vote from peers cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133), e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099)
I20260811 02:37:44.294492 14904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb5c5cbab64ed1a8c60dcfc0578bb7" candidate_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e039f61b46234dc88ed53dc615dfb572"
I20260811 02:37:44.294549 15123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fb5c5cbab64ed1a8c60dcfc0578bb7" candidate_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc87615c43047858efb37206c32fe1f"
I20260811 02:37:44.294579 14904 raft_consensus.cc:3037] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:44.294637 15123 raft_consensus.cc:3037] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:44.295358 14904 raft_consensus.cc:2454] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e29543d60f5459bb151b5cfcb4e64dc in term 1.
I20260811 02:37:44.295408 15123 raft_consensus.cc:2454] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e29543d60f5459bb151b5cfcb4e64dc in term 1.
I20260811 02:37:44.295542 14551 leader_election.cc:302] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e29543d60f5459bb151b5cfcb4e64dc, e039f61b46234dc88ed53dc615dfb572; no voters:
I20260811 02:37:44.295660 15280 raft_consensus.cc:2781] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:44.295791 15280 raft_consensus.cc:686] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 LEADER]: Becoming Leader. State: Replica: 0e29543d60f5459bb151b5cfcb4e64dc, State: Running, Role: LEADER
I20260811 02:37:44.295951 15280 consensus_queue.cc:237] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:44.297241 13104 catalog_manager.cc:5495] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1). New cstate: current_term: 1 leader_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } health_report { overall_health: HEALTHY } } }
W20260811 02:37:44.299122 14747 tablet.cc:2369] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:44.314232 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.310978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.2.251.1:34981)
0811 02:37:44.311075 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.311077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.311087 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:44.311135 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.311136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.311161 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.311231 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:44.311451 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.312030 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.312898 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.313268 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.313344 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.313651 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:44.313837 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:44.313936 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.314108 (+ 172us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.314140 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 02:37:44.316216 15123 raft_consensus.cc:1261] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Refusing update from remote peer 0e29543d60f5459bb151b5cfcb4e64dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:44.316217 14904 raft_consensus.cc:1261] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Refusing update from remote peer 0e29543d60f5459bb151b5cfcb4e64dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:44.316612 15281 consensus_queue.cc:1035] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:44.316751 15280 consensus_queue.cc:1035] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:44.319988 15288 mvcc.cc:205] Tried to move back new op lower bound from 7317159380236410880 to 7317159380156858368. Current Snapshot: MvccSnapshot[applied={T|T < 7317159380236410880}]
I20260811 02:37:44.323223 13104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:44.323418 13104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:44.325382 13104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7c088fc74e3491fb0b81d75400b7332: Sending DeleteTablet for 3 replicas of tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7
I20260811 02:37:44.325850 14866 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:44 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:39648
I20260811 02:37:44.325991 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:44 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:41670
I20260811 02:37:44.326134 14632 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fb5c5cbab64ed1a8c60dcfc0578bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:44 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:44520
I20260811 02:37:44.326207 15301 tablet_replica.cc:330] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: stopping tablet replica
I20260811 02:37:44.326325 15301 raft_consensus.cc:2229] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:44.326453 15301 raft_consensus.cc:2258] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:44.326457 15303 tablet_replica.cc:330] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: stopping tablet replica
I20260811 02:37:44.326653 15303 raft_consensus.cc:2229] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:44.326841 15303 raft_consensus.cc:2258] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:44.327322 15301 ts_tablet_manager.cc:1905] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:44.327565 15303 ts_tablet_manager.cc:1905] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:44.327816 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 13048
I20260811 02:37:44.334667 15303 ts_tablet_manager.cc:1918] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:44.334918 15303 log.cc:1196] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/f2fb5c5cbab64ed1a8c60dcfc0578bb7
I20260811 02:37:44.335355 15303 ts_tablet_manager.cc:1939] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P 0e29543d60f5459bb151b5cfcb4e64dc: Deleting consensus metadata
I20260811 02:37:44.341485 15301 ts_tablet_manager.cc:1918] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:44.341610 15301 log.cc:1196] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/f2fb5c5cbab64ed1a8c60dcfc0578bb7
I20260811 02:37:44.341920 15301 ts_tablet_manager.cc:1939] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P e039f61b46234dc88ed53dc615dfb572: Deleting consensus metadata
I20260811 02:37:44.348555 15302 tablet_replica.cc:330] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: stopping tablet replica
I20260811 02:37:44.348857 15302 raft_consensus.cc:2229] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:44.349077 15302 raft_consensus.cc:2258] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:44.349782 15302 ts_tablet_manager.cc:1905] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:44.352110 15302 ts_tablet_manager.cc:1918] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:44.352453 15302 log.cc:1196] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/f2fb5c5cbab64ed1a8c60dcfc0578bb7
I20260811 02:37:44.352855 15302 ts_tablet_manager.cc:1939] T f2fb5c5cbab64ed1a8c60dcfc0578bb7 P cfc87615c43047858efb37206c32fe1f: Deleting consensus metadata
W20260811 02:37:44.356381 14543 connection.cc:535] server connection from 127.0.0.1:44520 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 02:37:44.356413 14793 connection.cc:535] server connection from 127.0.0.1:39648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 02:37:44.356793 15010 connection.cc:535] server connection from 127.0.0.1:41670 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 02:37:44.357895 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.327552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.2.251.60:37315)
0811 02:37:44.327597 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.327599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.348357 (+ 20758us) server_negotiation.cc:407] Connection header received
0811 02:37:44.348449 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.348450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.348471 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.348560 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.349469 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.350388 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.351889 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.352280 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.352364 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.354426 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.354428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.354429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.354440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.356205 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.356636 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.356637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.356638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.356726 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.356944 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.356945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.356946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.357101 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.357200 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.357669 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.357825 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:44.366976 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.359758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.2.251.61:43011)
0811 02:37:44.359803 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.359805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.359817 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:37:44.360001 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.360002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.360025 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.360077 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.360950 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.361490 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.362311 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.362519 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.362579 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.364297 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.364299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.364299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.364307 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.365923 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.366242 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.366243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.366244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.366302 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.366467 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.366468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.366468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.366589 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.366682 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.366783 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.366876 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:37:44.370409 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:44.359583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.2.251.60:37315)
0811 02:37:44.359629 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:44.359631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:44.360112 (+ 481us) server_negotiation.cc:407] Connection header received
0811 02:37:44.360227 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:44.360228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:44.360241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:44.360272 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:44.361456 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.362422 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.364366 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:44.364813 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:44.364876 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:44.367480 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:44.367482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:44.367482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:44.367490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:44.368857 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.369504 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.369505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.369506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.369579 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:44.369842 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:44.369843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:44.369844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:44.370006 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:44.370094 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:44.370226 (+ 132us) server_negotiation.cc:299] Negotiation successful
0811 02:37:44.370311 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:45.321094 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.320735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:34463)
0811 02:37:45.320920 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.321032 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260811 02:37:45.321765 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.321114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:53883)
0811 02:37:45.321614 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.321692 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":53,"threads_started":1}
W20260811 02:37:45.321982 14744 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:41783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
W20260811 02:37:45.322093 14974 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:41783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:45.322129 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.321286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:49393)
0811 02:37:45.321969 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.322090 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":46,"threads_started":1}
W20260811 02:37:45.322528 15212 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:41783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:45.826139 15323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c7c088fc74e3491fb0b81d75400b7332)
I20260811 02:37:45.826145 15327 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c7c088fc74e3491fb0b81d75400b7332)
I20260811 02:37:45.826618 15323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:45.826707 15327 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:45.827059 15327 leader_election.cc:288] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783), 50baeab390ed42f3be30cfbae6840004 (127.2.251.60:37315)
I20260811 02:37:45.827930 15323 leader_election.cc:288] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783), a7126980727f4079a8447531f08e8269 (127.2.251.61:43011)
I20260811 02:37:45.828300 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.827352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42254)
0811 02:37:45.828111 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.828227 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":435,"threads_started":1}
I20260811 02:37:45.828369 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.828007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42258)
0811 02:37:45.828253 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.828326 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
W20260811 02:37:45.828486 14095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
W20260811 02:37:45.828539 14406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:45.829058 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.828953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42280)
0811 02:37:45.829009 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.829029 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260811 02:37:45.829058 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.828931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42270)
0811 02:37:45.828998 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.829029 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 02:37:45.829164 14406 leader_election.cc:334] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
W20260811 02:37:45.829519 14095 leader_election.cc:334] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:45.832814 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.827688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:43011 (local address 127.0.0.1:58810)
0811 02:37:45.827984 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.828002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:45.828241 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:45.828721 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:45.828722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:45.828726 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:45.828841 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:45.828844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.830166 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.830168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:45.831583 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:45.831589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.832503 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.832504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:45.832581 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:45.832587 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:45.832661 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 02:37:45.832715 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":72,"threads_started":1}
I20260811 02:37:45.832835 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.827630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37315 (local address 127.0.0.1:48030)
0811 02:37:45.828199 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.828211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:45.828281 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:45.828916 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:45.828917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:45.828920 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:45.828996 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:45.828997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.830166 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.830168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:45.831583 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:45.831589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.832502 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.832504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:45.832581 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:45.832587 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:45.832661 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 02:37:45.832714 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":69,"threads_started":1}
I20260811 02:37:45.832928 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.827875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.2.251.60:37315)
0811 02:37:45.828389 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:45.828391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:45.828400 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:45.828773 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:45.828774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:45.828786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:45.828874 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:45.829066 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.830007 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.831762 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.832378 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.832474 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:45.832492 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:45.832791 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 02:37:45.832852 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":46,"threads_started":1}
I20260811 02:37:45.832928 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.827846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.2.251.61:43011)
0811 02:37:45.828187 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:45.828189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:45.828203 (+ 14us) server_negotiation.cc:407] Connection header received
0811 02:37:45.828343 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:45.828344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:45.828356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:45.828612 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:45.828927 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.830007 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.831762 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:45.832371 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:45.832475 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:45.832492 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:45.832794 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 02:37:45.832852 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":319,"thread_start_us":61,"threads_started":1}
I20260811 02:37:45.833324 14137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50baeab390ed42f3be30cfbae6840004" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7126980727f4079a8447531f08e8269" is_pre_election: true
I20260811 02:37:45.833458 14137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50baeab390ed42f3be30cfbae6840004 in term 1.
I20260811 02:37:45.833402 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7126980727f4079a8447531f08e8269" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50baeab390ed42f3be30cfbae6840004" is_pre_election: true
I20260811 02:37:45.833526 14446 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7126980727f4079a8447531f08e8269 in term 1.
I20260811 02:37:45.833688 14404 leader_election.cc:302] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50baeab390ed42f3be30cfbae6840004, a7126980727f4079a8447531f08e8269; no voters: c7c088fc74e3491fb0b81d75400b7332
I20260811 02:37:45.833695 14094 leader_election.cc:302] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50baeab390ed42f3be30cfbae6840004, a7126980727f4079a8447531f08e8269; no voters: c7c088fc74e3491fb0b81d75400b7332
I20260811 02:37:45.833840 15327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 02:37:45.833846 15323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 02:37:45.833879 15323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c7c088fc74e3491fb0b81d75400b7332)
I20260811 02:37:45.833879 15327 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c7c088fc74e3491fb0b81d75400b7332)
I20260811 02:37:45.833901 15327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 1 FOLLOWER]: Advancing to term 2
I20260811 02:37:45.833901 15323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 1 FOLLOWER]: Advancing to term 2
I20260811 02:37:45.834676 15327 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:45.834692 15323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:45.834818 15323 leader_election.cc:288] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 2 election: Requested vote from peers c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783), a7126980727f4079a8447531f08e8269 (127.2.251.61:43011)
I20260811 02:37:45.834972 15327 leader_election.cc:288] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [CANDIDATE]: Term 2 election: Requested vote from peers c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783), 50baeab390ed42f3be30cfbae6840004 (127.2.251.60:37315)
I20260811 02:37:45.835083 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.834916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42296)
0811 02:37:45.834978 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.835049 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 02:37:45.835103 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.834970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42304)
0811 02:37:45.835057 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.835079 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 02:37:45.835120 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7126980727f4079a8447531f08e8269" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50baeab390ed42f3be30cfbae6840004"
I20260811 02:37:45.835114 14137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50baeab390ed42f3be30cfbae6840004" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7126980727f4079a8447531f08e8269"
I20260811 02:37:45.835210 14446 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a7126980727f4079a8447531f08e8269 in current term 2: Already voted for candidate 50baeab390ed42f3be30cfbae6840004 in this term.
I20260811 02:37:45.835209 14137 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 50baeab390ed42f3be30cfbae6840004 in current term 2: Already voted for candidate a7126980727f4079a8447531f08e8269 in this term.
I20260811 02:37:45.835431 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.835308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42320)
0811 02:37:45.835381 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.835407 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260811 02:37:45.835490 14406 leader_election.cc:334] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:45.835523 14406 leader_election.cc:302] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50baeab390ed42f3be30cfbae6840004; no voters: a7126980727f4079a8447531f08e8269, c7c088fc74e3491fb0b81d75400b7332
I20260811 02:37:45.835585 15323 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 02:37:45.835732 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:45.835613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42330)
0811 02:37:45.835678 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:45.835709 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 02:37:45.835817 14095 leader_election.cc:334] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:45.835852 14095 leader_election.cc:302] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7126980727f4079a8447531f08e8269; no voters: 50baeab390ed42f3be30cfbae6840004, c7c088fc74e3491fb0b81d75400b7332
I20260811 02:37:45.835922 15327 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 02:37:46.323228 15343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:46.322896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:57531)
0811 02:37:46.323122 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:46.323176 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260811 02:37:46.323226 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:46.322554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:33299)
0811 02:37:46.323092 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:46.323174 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":76,"threads_started":1}
I20260811 02:37:46.323226 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:46.322899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:46213)
0811 02:37:46.323091 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:46.323145 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260811 02:37:47.324330 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.323904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:53441)
0811 02:37:47.324191 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.324248 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":128,"threads_started":1}
I20260811 02:37:47.324424 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.323934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:49531)
0811 02:37:47.324331 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.324374 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":79,"threads_started":1}
I20260811 02:37:47.324734 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.324144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:53153)
0811 02:37:47.324600 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.324669 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":62,"threads_started":1}
I20260811 02:37:47.615617 15350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:47.615751 15350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:47.615958 15350 leader_election.cc:288] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783), a7126980727f4079a8447531f08e8269 (127.2.251.61:43011)
I20260811 02:37:47.616282 14137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50baeab390ed42f3be30cfbae6840004" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7126980727f4079a8447531f08e8269" is_pre_election: true
I20260811 02:37:47.616379 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.616134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42554)
0811 02:37:47.616291 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.616336 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260811 02:37:47.616413 14137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50baeab390ed42f3be30cfbae6840004 in term 2.
I20260811 02:37:47.616595 14404 leader_election.cc:302] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50baeab390ed42f3be30cfbae6840004, a7126980727f4079a8447531f08e8269; no voters:
I20260811 02:37:47.616699 15350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260811 02:37:47.616734 15350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:47.616755 15350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 2 FOLLOWER]: Advancing to term 3
I20260811 02:37:47.616858 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.616769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42564)
0811 02:37:47.616805 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.616835 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260811 02:37:47.616936 14406 leader_election.cc:334] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:47.617622 15350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:47.617878 15350 leader_election.cc:288] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 3 election: Requested vote from peers c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783), a7126980727f4079a8447531f08e8269 (127.2.251.61:43011)
I20260811 02:37:47.617997 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.617886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42568)
0811 02:37:47.617942 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.617971 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 02:37:47.617978 14137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50baeab390ed42f3be30cfbae6840004" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7126980727f4079a8447531f08e8269"
I20260811 02:37:47.618062 14137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 2 FOLLOWER]: Advancing to term 3
I20260811 02:37:47.618311 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.618217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42570)
0811 02:37:47.618269 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.618293 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 02:37:47.618386 14406 leader_election.cc:334] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
I20260811 02:37:47.618883 14137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50baeab390ed42f3be30cfbae6840004 in term 3.
I20260811 02:37:47.619295 14404 leader_election.cc:302] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50baeab390ed42f3be30cfbae6840004, a7126980727f4079a8447531f08e8269; no voters: c7c088fc74e3491fb0b81d75400b7332
I20260811 02:37:47.619544 15350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 02:37:47.619818 15350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [term 3 LEADER]: Becoming Leader. State: Replica: 50baeab390ed42f3be30cfbae6840004, State: Running, Role: LEADER
I20260811 02:37:47.619921 15350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } }
I20260811 02:37:47.620597 15355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50baeab390ed42f3be30cfbae6840004. Latest consensus state: current_term: 3 leader_uuid: "50baeab390ed42f3be30cfbae6840004" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:47.620776 15355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:47.621055 15360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:47.621752 15360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:47.621865 15360 catalog_manager.cc:1259] Loaded cluster ID: fa29a9b4fca3464db553767f6a8aa3e7
I20260811 02:37:47.621896 15360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:47.622114 15360 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:47.622202 15360 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Loaded TSK: 0
I20260811 02:37:47.622354 15360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:37:47.631476 14434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:47.633213 14434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:47.634656 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.634530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42584)
0811 02:37:47.634594 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.634626 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 02:37:47.634788 14137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [term 3 FOLLOWER]: Refusing update from remote peer 50baeab390ed42f3be30cfbae6840004: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260811 02:37:47.635015 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.634923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.0.0.1:42586)
0811 02:37:47.634958 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.634990 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260811 02:37:47.635058 15355 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260811 02:37:47.635192 14406 consensus_peers.cc:487] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 -> Peer c7c088fc74e3491fb0b81d75400b7332 (127.2.251.62:41783): Couldn't send request to peer c7c088fc74e3491fb0b81d75400b7332. Status: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 02:37:47.635994 15363 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50baeab390ed42f3be30cfbae6840004. Latest consensus state: current_term: 3 leader_uuid: "50baeab390ed42f3be30cfbae6840004" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:47.636106 15363 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:37:47.636770 15355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "50baeab390ed42f3be30cfbae6840004" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:47.636868 15355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:47.636965 15364 mvcc.cc:205] Tried to move back new op lower bound from 7317159393828732928 to 7317159393772343296. Current Snapshot: MvccSnapshot[applied={T|T < 7317159393828732928}]
I20260811 02:37:47.637080 15363 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "50baeab390ed42f3be30cfbae6840004" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c7c088fc74e3491fb0b81d75400b7332" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41783 } } peers { permanent_uuid: "a7126980727f4079a8447531f08e8269" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 43011 } attrs { promote: false } } peers { permanent_uuid: "50baeab390ed42f3be30cfbae6840004" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37315 } attrs { promote: false } } }
I20260811 02:37:47.637176 15363 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7126980727f4079a8447531f08e8269 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:37:47.643772 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.638961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:43099 (local address 127.0.0.1:39662)
0811 02:37:47.639163 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.639195 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:47.639414 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:47.640095 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:47.640096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:47.640100 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:47.640191 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.640194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.641665 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.641666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.642627 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.642630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.643445 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.643446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.643513 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.643521 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 02:37:47.643600 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 02:37:47.643662 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260811 02:37:47.643814 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.639318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.2.251.2:43099)
0811 02:37:47.639942 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:47.639944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:47.639954 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:47.639987 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:47.639988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:47.639999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:47.640085 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:47.640288 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.641404 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.642738 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.643340 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.643428 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.643446 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:47.643622 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 02:37:47.643685 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":147,"threads_started":1}
I20260811 02:37:47.643918 15351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.639009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.0.0.1:44544)
0811 02:37:47.639502 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.639516 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:47.639573 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:47.639996 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:47.639998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:47.640002 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:47.640138 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.640142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.641576 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.641578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.642829 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.642832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.643661 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.643662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.643735 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.643742 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 02:37:47.643772 (+ 30us) client_negotiation.cc:240] Negotiation successful
0811 02:37:47.643841 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":116,"thread_start_us":92,"threads_started":1}
I20260811 02:37:47.644284 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.638912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:41133 (local address 127.0.0.1:41686)
0811 02:37:47.639159 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.639171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:47.639230 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:47.640218 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:47.640219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:47.640221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:47.640264 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.640266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.641393 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.641395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.643177 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.643180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.643992 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.643993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.644068 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.644074 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:47.644133 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 02:37:47.644187 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":4}
I20260811 02:37:47.644320 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.639174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.2.251.3:41133)
0811 02:37:47.640032 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:47.640036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:47.640047 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:47.640079 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:47.640081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:47.640094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:47.640205 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:47.640331 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.641255 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.643280 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.643856 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.643932 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.643948 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:47.644156 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 02:37:47.644241 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":206,"threads_started":1}
I20260811 02:37:47.644301 14866 tablet_service.cc:1467] Processing CreateTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 (DEFAULT_TABLE table=test-table [id=4f35d691046d4a5ea708e0c80c6652cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:47.644516 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50162eb5021042ba8e620fa2d75b2ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:47.645210 15091 tablet_service.cc:1467] Processing CreateTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 (DEFAULT_TABLE table=test-table [id=4f35d691046d4a5ea708e0c80c6652cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:47.645404 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50162eb5021042ba8e620fa2d75b2ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:47.646196 15378 tablet_bootstrap.cc:492] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Bootstrap starting.
I20260811 02:37:47.646413 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.639502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.2.251.1:34981)
0811 02:37:47.639757 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:47.639760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:47.639773 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:47.639816 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:47.639817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:47.639831 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:47.639950 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:47.640364 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.641410 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.642937 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.643552 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.646195 (+ 2643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.646227 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:47.646250 (+ 23us) server_negotiation.cc:299] Negotiation successful
0811 02:37:47.646325 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260811 02:37:47.646787 15378 tablet_bootstrap.cc:654] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:47.646796 14632 tablet_service.cc:1467] Processing CreateTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 (DEFAULT_TABLE table=test-table [id=4f35d691046d4a5ea708e0c80c6652cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:47.646971 14632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50162eb5021042ba8e620fa2d75b2ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:47.647660 15382 tablet_bootstrap.cc:492] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Bootstrap starting.
I20260811 02:37:47.647722 15378 tablet_bootstrap.cc:492] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: No bootstrap required, opened a new log
I20260811 02:37:47.647766 15378 ts_tablet_manager.cc:1397] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:47.647971 15378 raft_consensus.cc:348] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.648059 15378 raft_consensus.cc:374] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:47.648082 15378 raft_consensus.cc:729] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e039f61b46234dc88ed53dc615dfb572, State: Initialized, Role: FOLLOWER
I20260811 02:37:47.648142 15378 consensus_queue.cc:260] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.648363 15378 ts_tablet_manager.cc:1428] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:47.648738 15385 tablet_bootstrap.cc:492] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: Bootstrap starting.
I20260811 02:37:47.649355 15385 tablet_bootstrap.cc:654] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:47.649360 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.649204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:57661)
0811 02:37:47.649275 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.649317 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 02:37:47.649818 14969 heartbeater.cc:503] Master 127.2.251.60:37315 was elected leader, sending a full tablet report...
I20260811 02:37:47.649827 15382 tablet_bootstrap.cc:654] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:47.650187 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.650097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:45943)
0811 02:37:47.650134 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.650160 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260811 02:37:47.650228 15385 tablet_bootstrap.cc:492] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: No bootstrap required, opened a new log
I20260811 02:37:47.650293 15385 ts_tablet_manager.cc:1397] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:47.650436 15385 raft_consensus.cc:348] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.650509 15385 raft_consensus.cc:374] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:47.650528 15385 raft_consensus.cc:729] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e29543d60f5459bb151b5cfcb4e64dc, State: Initialized, Role: FOLLOWER
I20260811 02:37:47.650573 15385 consensus_queue.cc:260] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.650764 15385 ts_tablet_manager.cc:1428] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:47.651026 15382 tablet_bootstrap.cc:492] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: No bootstrap required, opened a new log
I20260811 02:37:47.651093 15382 ts_tablet_manager.cc:1397] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260811 02:37:47.651245 14742 heartbeater.cc:503] Master 127.2.251.60:37315 was elected leader, sending a full tablet report...
I20260811 02:37:47.651263 15382 raft_consensus.cc:348] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.651348 15382 raft_consensus.cc:374] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:47.651369 15382 raft_consensus.cc:729] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc87615c43047858efb37206c32fe1f, State: Initialized, Role: FOLLOWER
I20260811 02:37:47.651425 15382 consensus_queue.cc:260] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.651633 15382 ts_tablet_manager.cc:1428] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:47.651849 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.650952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:32909)
0811 02:37:47.651781 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.651816 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":810}
I20260811 02:37:47.652099 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.652002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:59821)
0811 02:37:47.652042 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.652071 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260811 02:37:47.652396 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.652309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:36781)
0811 02:37:47.652348 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.652370 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260811 02:37:47.652544 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.652324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:38823)
0811 02:37:47.652483 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.652514 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260811 02:37:47.682950 15216 tablet.cc:2369] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:47.883849 15393 raft_consensus.cc:482] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:47.884012 15393 raft_consensus.cc:504] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.884580 15393 leader_election.cc:288] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099), 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981)
I20260811 02:37:47.887849 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.885046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:56799 (local address 127.2.251.1:34981)
0811 02:37:47.885100 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:47.885101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:47.885111 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:47.885148 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:47.885149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:47.885161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:47.885213 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:47.885375 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.885993 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.887104 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.887507 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.887576 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.887595 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:47.887722 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 02:37:47.887763 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:37:47.887878 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.884838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.2.251.3:56799)
0811 02:37:47.884899 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.884914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:47.884996 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:47.885230 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:47.885231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:47.885234 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:47.885313 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.885315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.886094 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.886095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.887003 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.887005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.887592 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.887593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.887652 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.887657 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:47.887704 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:47.887760 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":4}
I20260811 02:37:47.887918 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.884949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:60205 (local address 127.2.251.2:43099)
0811 02:37:47.885053 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:47.885057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:47.885319 (+ 262us) server_negotiation.cc:407] Connection header received
0811 02:37:47.885375 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:47.885376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:47.885391 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:47.885441 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:47.885572 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.886109 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.887156 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.887550 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.887613 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.887629 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:47.887792 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 02:37:47.887845 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 02:37:47.887954 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.884845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:43099 (local address 127.2.251.3:60205)
0811 02:37:47.885230 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.885239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:47.885314 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:47.885442 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:47.885442 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:47.885444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:47.885504 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.885506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.886193 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.886194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.886997 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:47.886999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.887658 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.887660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:47.887720 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.887725 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:37:47.887772 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:47.887819 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":119,"thread_start_us":184,"threads_started":1}
I20260811 02:37:47.888267 14904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50162eb5021042ba8e620fa2d75b2ca9" candidate_uuid: "cfc87615c43047858efb37206c32fe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e039f61b46234dc88ed53dc615dfb572" is_pre_election: true
I20260811 02:37:47.888403 14904 raft_consensus.cc:2454] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc87615c43047858efb37206c32fe1f in term 0.
I20260811 02:37:47.888458 14662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50162eb5021042ba8e620fa2d75b2ca9" candidate_uuid: "cfc87615c43047858efb37206c32fe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" is_pre_election: true
I20260811 02:37:47.888603 15014 leader_election.cc:302] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc87615c43047858efb37206c32fe1f, e039f61b46234dc88ed53dc615dfb572; no voters:
I20260811 02:37:47.888648 14662 raft_consensus.cc:2454] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfc87615c43047858efb37206c32fe1f in term 0.
I20260811 02:37:47.888784 15393 raft_consensus.cc:2781] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:47.888831 15393 raft_consensus.cc:482] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:47.888875 15393 raft_consensus.cc:3037] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:47.889871 15393 raft_consensus.cc:504] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.890017 15393 leader_election.cc:288] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [CANDIDATE]: Term 1 election: Requested vote from peers e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099), 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981)
I20260811 02:37:47.890441 14662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50162eb5021042ba8e620fa2d75b2ca9" candidate_uuid: "cfc87615c43047858efb37206c32fe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e29543d60f5459bb151b5cfcb4e64dc"
I20260811 02:37:47.890450 14904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50162eb5021042ba8e620fa2d75b2ca9" candidate_uuid: "cfc87615c43047858efb37206c32fe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e039f61b46234dc88ed53dc615dfb572"
I20260811 02:37:47.890565 14662 raft_consensus.cc:3037] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:47.890579 14904 raft_consensus.cc:3037] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:47.891377 14904 raft_consensus.cc:2454] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc87615c43047858efb37206c32fe1f in term 1.
I20260811 02:37:47.891407 14662 raft_consensus.cc:2454] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfc87615c43047858efb37206c32fe1f in term 1.
I20260811 02:37:47.891628 15014 leader_election.cc:302] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc87615c43047858efb37206c32fe1f, e039f61b46234dc88ed53dc615dfb572; no voters:
I20260811 02:37:47.891790 15393 raft_consensus.cc:2781] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:47.891985 15393 raft_consensus.cc:686] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 1 LEADER]: Becoming Leader. State: Replica: cfc87615c43047858efb37206c32fe1f, State: Running, Role: LEADER
I20260811 02:37:47.892059 15210 heartbeater.cc:503] Master 127.2.251.60:37315 was elected leader, sending a full tablet report...
I20260811 02:37:47.892131 15393 consensus_queue.cc:237] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } }
I20260811 02:37:47.892385 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.892231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:60487)
0811 02:37:47.892301 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.892349 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 02:37:47.893059 14434 catalog_manager.cc:5495] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to cfc87615c43047858efb37206c32fe1f (127.2.251.3). New cstate: current_term: 1 leader_uuid: "cfc87615c43047858efb37206c32fe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc87615c43047858efb37206c32fe1f" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:47.935328 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.931937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.2.251.3:41133)
0811 02:37:47.932064 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:47.932066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:47.932076 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:47.932119 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:47.932120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:47.932144 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:47.932216 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:47.932474 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.933039 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.933963 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:47.934235 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:47.934342 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:47.934736 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:47.934875 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:47.934994 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:47.935217 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 02:37:47.935256 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 02:37:47.936554 14904 raft_consensus.cc:1261] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Refusing update from remote peer cfc87615c43047858efb37206c32fe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:47.936614 14662 raft_consensus.cc:1261] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Refusing update from remote peer cfc87615c43047858efb37206c32fe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:47.936878 15398 consensus_queue.cc:1035] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e039f61b46234dc88ed53dc615dfb572" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:47.937083 15393 consensus_queue.cc:1035] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e29543d60f5459bb151b5cfcb4e64dc" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:47.937659 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.937541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:48361)
0811 02:37:47.937591 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.937624 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 02:37:47.938126 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.938021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:33109)
0811 02:37:47.938064 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.938091 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 02:37:47.938963 15405 mvcc.cc:205] Tried to move back new op lower bound from 7317159395065835520 to 7317159394886991872. Current Snapshot: MvccSnapshot[applied={T|T < 7317159395065835520}]
I20260811 02:37:47.939436 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.939212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:41033)
0811 02:37:47.939374 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.939407 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260811 02:37:47.940058 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.939971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.3:34195)
0811 02:37:47.940007 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.940031 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260811 02:37:47.940156 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.940053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.2:45999)
0811 02:37:47.940095 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.940126 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260811 02:37:47.941787 14430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:47.941891 14430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:47.942463 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.942355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41783 (local address 127.2.251.1:51711)
0811 02:37:47.942401 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.942427 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:41783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260811 02:37:47.944489 14430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50baeab390ed42f3be30cfbae6840004: Sending DeleteTablet for 3 replicas of tablet 50162eb5021042ba8e620fa2d75b2ca9
I20260811 02:37:47.944777 14632 tablet_service.cc:1514] Processing DeleteTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:47 UTC) from {username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:44544
I20260811 02:37:47.945183 14866 tablet_service.cc:1514] Processing DeleteTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:47 UTC) from {username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:39662
I20260811 02:37:47.945358 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:47 UTC) from {username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:41686
I20260811 02:37:47.945546 15425 tablet_replica.cc:330] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: stopping tablet replica
I20260811 02:37:47.945561 15424 tablet_replica.cc:330] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: stopping tablet replica
I20260811 02:37:47.945621 15425 raft_consensus.cc:2229] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:47.945626 15424 raft_consensus.cc:2229] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:47.945685 15425 raft_consensus.cc:2258] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:47.945679 14632 tablet_service.cc:1514] Processing DeleteTablet for tablet 50162eb5021042ba8e620fa2d75b2ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:47 UTC) from {username='oryx', principal='oryx/127.2.251.60@KRBTEST.COM'} at 127.0.0.1:44544
I20260811 02:37:47.945685 15424 raft_consensus.cc:2258] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:47.946492 15424 ts_tablet_manager.cc:1905] T 50162eb5021042ba8e620fa2d75b2ca9 P 0e29543d60f5459bb151b5cfcb4e64dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:47.946516 15425 ts_tablet_manager.cc:1905] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:47.946560 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 14355
I20260811 02:37:47.948350 15425 ts_tablet_manager.cc:1918] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:47.948406 15425 log.cc:1196] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/50162eb5021042ba8e620fa2d75b2ca9
I20260811 02:37:47.948562 15425 ts_tablet_manager.cc:1939] T 50162eb5021042ba8e620fa2d75b2ca9 P e039f61b46234dc88ed53dc615dfb572: Deleting consensus metadata
I20260811 02:37:47.948915 14408 catalog_manager.cc:4841] TS e039f61b46234dc88ed53dc615dfb572 (127.2.251.2:43099): tablet 50162eb5021042ba8e620fa2d75b2ca9 (table test-table [id=4f35d691046d4a5ea708e0c80c6652cc]) successfully deleted
I20260811 02:37:47.961530 15426 tablet_replica.cc:330] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: stopping tablet replica
I20260811 02:37:47.961769 15426 raft_consensus.cc:2229] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:47.961988 15426 raft_consensus.cc:2258] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:47.965893 15426 ts_tablet_manager.cc:1905] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:47.968122 15426 ts_tablet_manager.cc:1918] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:47.968204 15426 log.cc:1196] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/50162eb5021042ba8e620fa2d75b2ca9
I20260811 02:37:47.968431 15426 ts_tablet_manager.cc:1939] T 50162eb5021042ba8e620fa2d75b2ca9 P cfc87615c43047858efb37206c32fe1f: Deleting consensus metadata
I20260811 02:37:47.969136 14404 catalog_manager.cc:4841] TS cfc87615c43047858efb37206c32fe1f (127.2.251.3:41133): tablet 50162eb5021042ba8e620fa2d75b2ca9 (table test-table [id=4f35d691046d4a5ea708e0c80c6652cc]) successfully deleted
I20260811 02:37:47.974355 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 14753
I20260811 02:37:47.974489 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:47.974347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.0.0.1:44550)
0811 02:37:47.974409 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:47.974444 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 02:37:47.974682 14404 catalog_manager.cc:4584] TS 0e29543d60f5459bb151b5cfcb4e64dc (127.2.251.1:34981): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50162eb5021042ba8e620fa2d75b2ca9: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34981: connect: Connection refused (error 111)
I20260811 02:37:47.989050 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 14981
I20260811 02:37:48.010279 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 13889
I20260811 02:37:48.026609 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:48.026452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.0.0.1:44556)
0811 02:37:48.026528 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:48.026564 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 02:37:48.027206 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:48.026949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.0.0.1:44562)
0811 02:37:48.027086 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:48.027158 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260811 02:37:48.027696 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:48.027548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34981 (local address 127.0.0.1:44568)
0811 02:37:48.027626 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:48.027667 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 02:37:48.030333 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 14209
2026-08-11T02:37:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7294 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:48 dist-test-slave-cvwc krb5kdc[15429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:48 dist-test-slave-cvwc krb5kdc[15429](info): set up 2 sockets
Aug 11 02:37:48 dist-test-slave-cvwc krb5kdc[15429](info): commencing operation
krb5kdc: starting...
W20260811 02:37:49.316480 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:49 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab.
Entry for principal oryx/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab.
I20260811 02:37:49.457181 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39355
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab
--principal=oryx/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:49.554340 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:49.554576 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:49.554613 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:49.556012 15451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:49.556085 15451 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 02:37:49.556108 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:49.556126 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:49.556141 15451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:49.556157 15451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:49.557958 15451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39355
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.62.keytab
--principal=oryx/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:49.558236 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:49.558490 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:49.564298 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:49.564319 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:49.564320 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:49.564410 15451 server_base.cc:1034] running on GCE node
I20260811 02:37:49.564827 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:49.565215 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:49.566396 15451 hybrid_clock.cc:648] HybridClock initialized: now 1786415869566364 us; error 52 us; skew 500 ppm
Aug 11 02:37:49 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:49.569413 15451 init.cc:377] Logged in from keytab as oryx/127.2.251.62@KRBTEST.COM (short username oryx)
I20260811 02:37:49.570220 15451 webserver.cc:460] Webserver started at http://127.2.251.62:41611/ using document root <none> and password file <none>
I20260811 02:37:49.570497 15451 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:49.570552 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:49.570686 15451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:49.571820 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "1f11a5203ee9499988df4ff4d2cd5f7c"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "be31a5b554e29691c309a2c2596c5756"
server_key_iv: "9b7e7bcf85cbc672fb78faddc468518f"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.572175 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "1f11a5203ee9499988df4ff4d2cd5f7c"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "be31a5b554e29691c309a2c2596c5756"
server_key_iv: "9b7e7bcf85cbc672fb78faddc468518f"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.573550 15451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:37:49.574448 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:49.574654 15451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:49.574746 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "1f11a5203ee9499988df4ff4d2cd5f7c"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "be31a5b554e29691c309a2c2596c5756"
server_key_iv: "9b7e7bcf85cbc672fb78faddc468518f"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.574812 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:49.587625 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:49.589964 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:49.590113 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:49.594794 15451 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39355
I20260811 02:37:49.594856 15582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39355 every 8 connection(s)
I20260811 02:37:49.595218 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:49.595888 15584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:49.598273 15584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Bootstrap starting.
I20260811 02:37:49.598910 15584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:49.599184 15584 log.cc:824] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:49.599923 15584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: No bootstrap required, opened a new log
I20260811 02:37:49.602109 15584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39355 } }
I20260811 02:37:49.602272 15584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:49.602298 15584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f11a5203ee9499988df4ff4d2cd5f7c, State: Initialized, Role: FOLLOWER
I20260811 02:37:49.602437 15584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39355 } }
I20260811 02:37:49.602509 15584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:49.602531 15584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:49.602685 15584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:49.603310 15584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39355 } }
I20260811 02:37:49.603420 15584 leader_election.cc:302] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f11a5203ee9499988df4ff4d2cd5f7c; no voters:
I20260811 02:37:49.603600 15584 leader_election.cc:288] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:49.603681 15589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:49.603888 15584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:49.603912 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 15451
I20260811 02:37:49.603995 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:49.603921 15589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [term 1 LEADER]: Becoming Leader. State: Replica: 1f11a5203ee9499988df4ff4d2cd5f7c, State: Running, Role: LEADER
I20260811 02:37:49.604117 15589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39355 } }
I20260811 02:37:49.604218 3052 external_mini_cluster.cc:1428] Setting key 941b8f9f7ec8bcbbe92388e873467d7c
I20260811 02:37:49.604512 15590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39355 } } }
I20260811 02:37:49.604667 15590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:49.605157 15607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:49.605541 15591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f11a5203ee9499988df4ff4d2cd5f7c. Latest consensus state: current_term: 1 leader_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f11a5203ee9499988df4ff4d2cd5f7c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39355 } } }
I20260811 02:37:49.605659 15591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:49.605774 15607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:49.607722 15607 catalog_manager.cc:1347] Generated new cluster ID: 480aa1d1782e4c5a979c7069d8507f5a
I20260811 02:37:49.607826 15607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:37:49 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:49.617115 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:49.604899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.2.251.62:39355)
0811 02:37:49.605074 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:49.605086 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:49.605208 (+ 122us) server_negotiation.cc:407] Connection header received
0811 02:37:49.605911 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:49.605920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:49.605982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:49.606139 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:49.606915 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.608355 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.609437 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.610130 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.610260 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:49.613162 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:49.613166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:49.613167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:49.613183 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:49.615272 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:49.615783 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:49.615787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:49.615788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:49.615904 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:49.616099 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:49.616100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:49.616100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:49.616427 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:49.616579 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:49.616679 (+ 100us) server_negotiation.cc:299] Negotiation successful
0811 02:37:49.616849 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260811 02:37:49.640692 15607 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:49.641309 15607 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:49.646442 15607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Generated new TSK 0
I20260811 02:37:49.646649 15607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab.
Entry for principal oryx/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab.
I20260811 02:37:49.682168 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39355
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:49.782196 15624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:49.782440 15624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:49.782469 15624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:49.783638 15624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:49.783692 15624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:49.783721 15624 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:49.783741 15624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:49.785362 15624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:49.785604 15624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:49.785854 15624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:49.791635 15638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:49.791805 15624 server_base.cc:1034] running on GCE node
W20260811 02:37:49.791635 15633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:49.791879 15630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:49.792181 15624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:49.792445 15624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:49.793607 15624 hybrid_clock.cc:648] HybridClock initialized: now 1786415869793583 us; error 37 us; skew 500 ppm
Aug 11 02:37:49 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:49.796430 15624 init.cc:377] Logged in from keytab as oryx/127.2.251.1@KRBTEST.COM (short username oryx)
I20260811 02:37:49.796880 15624 webserver.cc:460] Webserver started at http://127.2.251.1:38787/ using document root <none> and password file <none>
I20260811 02:37:49.797104 15624 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:49.797147 15624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:49.797243 15624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:49.798385 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "6b6a94b43ef342d7be97d55be87dc2b6"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "94df79de56daf79f56d1bfd686e94ab9"
server_key_iv: "c55605a6c8decdc42bd1a6a4c1674389"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.798718 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "6b6a94b43ef342d7be97d55be87dc2b6"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "94df79de56daf79f56d1bfd686e94ab9"
server_key_iv: "c55605a6c8decdc42bd1a6a4c1674389"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.800329 15624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:49.801265 15647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:49.801568 15624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:49.801666 15624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "6b6a94b43ef342d7be97d55be87dc2b6"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "94df79de56daf79f56d1bfd686e94ab9"
server_key_iv: "c55605a6c8decdc42bd1a6a4c1674389"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.801781 15624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:49.816205 15624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:49.819136 15624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:49.819288 15624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:49.819913 15624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:49.819958 15624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:49.819985 15624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:49.820000 15624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:49.828871 15624 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:37057
I20260811 02:37:49.828997 15841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:37057 every 8 connection(s)
Aug 11 02:37:49 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:49.829393 15624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:49.830485 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 15624
I20260811 02:37:49.830601 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:49.830824 3052 external_mini_cluster.cc:1428] Setting key bef553f47cf0ddb57cfb95fcacc36093
I20260811 02:37:49.833406 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:49.820594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:50111 (local address 127.2.251.62:39355)
0811 02:37:49.820668 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:49.820670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:49.820905 (+ 235us) server_negotiation.cc:407] Connection header received
0811 02:37:49.821707 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:49.821738 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:49.821756 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:49.821832 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:49.822834 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.824439 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.826195 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.826544 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.826630 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:49.830199 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:49.830201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:49.830202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:49.830218 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:49.831980 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:49.832438 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:49.832439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:49.832439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:49.832496 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:49.832732 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:49.832733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:49.832733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:49.832854 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:49.832914 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:49.833203 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 02:37:49.833298 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:37:49.833595 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:49.820546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.2.251.1:50111)
0811 02:37:49.820746 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:49.820800 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:49.821545 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:49.821949 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:49.821954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:49.822380 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:49.822722 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:49.822731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.824655 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.824656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:49.826067 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:49.826072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.826644 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.826645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:49.826833 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:49.827701 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:49.827717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:49.830064 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:49.832069 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:49.832080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:49.832083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:49.832360 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:49.832553 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:49.832554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:49.832554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:49.832673 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:49.832914 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:49.832922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:49.833075 (+ 153us) client_negotiation.cc:769] Sending connection context
0811 02:37:49.833171 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 02:37:49.833357 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260811 02:37:49.834190 15843 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39355
I20260811 02:37:49.834326 15843 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:49.834545 15843 heartbeater.cc:511] Master 127.2.251.62:39355 requested a full tablet report, sending...
I20260811 02:37:49.835151 15496 ts_manager.cc:194] Registered new tserver with Master: 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057)
I20260811 02:37:49.835775 15496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:50111
WARNING: no policy specified for oryx/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.2@KRBTEST.COM" created.
I20260811 02:37:49.845399 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:49.838142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:57135 (local address 127.2.251.62:39355)
0811 02:37:49.838194 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:49.838196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:49.838329 (+ 133us) server_negotiation.cc:407] Connection header received
0811 02:37:49.838367 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:49.838368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:49.838382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:49.838453 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:49.838870 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.839650 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.840568 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.840860 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.840956 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:49.842077 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:49.842078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:49.842078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:49.842086 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:49.843503 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:49.843992 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:49.843994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:49.843994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:49.844057 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:49.844384 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:49.844385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:49.844386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:49.844509 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:49.844568 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:49.845238 (+ 670us) server_negotiation.cc:299] Negotiation successful
0811 02:37:49.845327 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:37:49.845409 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:49.838014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.2.251.1:57135)
0811 02:37:49.838246 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:49.838256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:49.838319 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:49.838494 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:49.838495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:49.838710 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:49.838770 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:49.838772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.839777 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.839778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:49.840471 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:49.840475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:49.840977 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:49.840978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:49.841051 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:49.841598 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:49.841603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:49.841951 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:49.843606 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:49.843607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:49.843608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:49.843902 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:49.844150 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:49.844151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:49.844152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:49.844247 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:49.845038 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:49.845039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:49.845101 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 02:37:49.845180 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 02:37:49.845310 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab.
Entry for principal oryx/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab.
I20260811 02:37:49.861662 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39355
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:49.950909 15856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:49.951182 15856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:49.951232 15856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:49.952621 15856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:49.952693 15856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:49.952728 15856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:49.952808 15856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:49.955704 15856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:49.956070 15856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:49.956321 15856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:49.962262 15866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:49.962275 15865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:49.962263 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:49.962705 15856 server_base.cc:1034] running on GCE node
I20260811 02:37:49.962939 15856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:49.963178 15856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:49.964339 15856 hybrid_clock.cc:648] HybridClock initialized: now 1786415869964325 us; error 34 us; skew 500 ppm
Aug 11 02:37:49 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:49.967329 15856 init.cc:377] Logged in from keytab as oryx/127.2.251.2@KRBTEST.COM (short username oryx)
I20260811 02:37:49.967865 15856 webserver.cc:460] Webserver started at http://127.2.251.2:43673/ using document root <none> and password file <none>
I20260811 02:37:49.968142 15856 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:49.968199 15856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:49.968344 15856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:49.969480 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "51330a0484270ea7f0a62f8b09d21284"
server_key_iv: "f3b967e48189b572a518b3dd6c56cbb9"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.969880 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "51330a0484270ea7f0a62f8b09d21284"
server_key_iv: "f3b967e48189b572a518b3dd6c56cbb9"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.971275 15856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:49.972461 15880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:49.972730 15856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:49.972834 15856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
format_stamp: "Formatted at 2026-08-11 02:37:49 on dist-test-slave-cvwc"
server_key: "51330a0484270ea7f0a62f8b09d21284"
server_key_iv: "f3b967e48189b572a518b3dd6c56cbb9"
server_key_version: "encryptionkey@0"
I20260811 02:37:49.972904 15856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:50.022863 15856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:50.025614 15856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:50.025835 15856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:50.026571 15856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:50.026620 15856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.026655 15856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:50.026672 15856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:50.034304 15856 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:39291
I20260811 02:37:50.034557 16070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:39291 every 8 connection(s)
I20260811 02:37:50.034771 15856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:50.037477 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.026980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52731 (local address 127.2.251.62:39355)
0811 02:37:50.027054 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.027056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.027279 (+ 223us) server_negotiation.cc:407] Connection header received
0811 02:37:50.027826 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.027828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.027842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.027922 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.028849 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.029593 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.030954 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.031251 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.031354 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.034451 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.034453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.034454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.034463 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.036066 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.036493 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.036495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.036495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.036559 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.036796 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.036797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.036798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.036937 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.036995 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.037285 (+ 290us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.037383 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 02:37:50.037674 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.026944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.2.251.2:52731)
0811 02:37:50.027119 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.027164 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.027724 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.027989 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.027993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.028387 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:50.028699 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.028707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.029756 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.029759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.030826 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.030832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.031391 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.031392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.031581 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.032420 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:50.032431 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:50.034343 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:50.036163 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.036171 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.036173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.036418 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.036617 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.036618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.036618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.036738 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.036999 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:50.037006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:50.037158 (+ 152us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.037252 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.037446 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260811 02:37:50.038360 16071 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39355
I20260811 02:37:50.038482 16071 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:50.038722 16071 heartbeater.cc:511] Master 127.2.251.62:39355 requested a full tablet report, sending...
I20260811 02:37:50.038903 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 15856
I20260811 02:37:50.039008 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:50.039217 3052 external_mini_cluster.cc:1428] Setting key 7b19202eae0d248dda8c05a123f838ae
I20260811 02:37:50.039214 15496 ts_manager.cc:194] Registered new tserver with Master: eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291)
I20260811 02:37:50.039707 15496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:52731
I20260811 02:37:50.046320 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.039953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:39969 (local address 127.2.251.62:39355)
0811 02:37:50.039997 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.039999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.040198 (+ 199us) server_negotiation.cc:407] Connection header received
0811 02:37:50.040280 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.040281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.040295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.040355 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.040775 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.041273 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.042247 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.042456 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.042515 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.043473 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.043475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.043475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.043482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.045104 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.045480 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.045482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.045482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.045540 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.045738 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.045739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.045740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.045914 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.045977 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.046129 (+ 152us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.046215 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:37:50.046336 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.039962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.2.251.2:39969)
0811 02:37:50.040125 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.040139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.040209 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.040365 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.040366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.040594 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:50.040679 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.040682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.041384 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.041385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.042161 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.042164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.042522 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.042523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.042586 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.043046 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:50.043051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:50.043373 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:50.045196 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.045197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.045197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.045408 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.045596 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.045597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.045597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.045664 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.045980 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:50.045982 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:50.046036 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.046097 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.046218 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab.
Entry for principal oryx/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab.
I20260811 02:37:50.073011 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39355
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:50.166350 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:50.166600 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:50.166640 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:50.167909 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:50.167969 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:50.168002 16085 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 02:37:50.168032 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:50.170753 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.251.62:39355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:50.171065 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:50.171319 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:50.176872 16096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:50.176889 16093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:50.177073 16085 server_base.cc:1034] running on GCE node
W20260811 02:37:50.176918 16092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:50.177423 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:50.177677 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:50.178890 16085 hybrid_clock.cc:648] HybridClock initialized: now 1786415870178870 us; error 33 us; skew 500 ppm
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415870, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:50.181569 16085 init.cc:377] Logged in from keytab as oryx/127.2.251.3@KRBTEST.COM (short username oryx)
I20260811 02:37:50.182272 16085 webserver.cc:460] Webserver started at http://127.2.251.3:35391/ using document root <none> and password file <none>
I20260811 02:37:50.182614 16085 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:50.182677 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:50.182796 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:50.183928 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "54f2be9273ed44348d6aa3b52d1a0604"
format_stamp: "Formatted at 2026-08-11 02:37:50 on dist-test-slave-cvwc"
server_key: "fc937f94fd72984aaf37b98281a11c36"
server_key_iv: "a44083fdca4db6e0704ac0435f6ead91"
server_key_version: "encryptionkey@0"
I20260811 02:37:50.184262 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "54f2be9273ed44348d6aa3b52d1a0604"
format_stamp: "Formatted at 2026-08-11 02:37:50 on dist-test-slave-cvwc"
server_key: "fc937f94fd72984aaf37b98281a11c36"
server_key_iv: "a44083fdca4db6e0704ac0435f6ead91"
server_key_version: "encryptionkey@0"
I20260811 02:37:50.185633 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260811 02:37:50.186496 16111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.186707 16085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.186807 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "54f2be9273ed44348d6aa3b52d1a0604"
format_stamp: "Formatted at 2026-08-11 02:37:50 on dist-test-slave-cvwc"
server_key: "fc937f94fd72984aaf37b98281a11c36"
server_key_iv: "a44083fdca4db6e0704ac0435f6ead91"
server_key_version: "encryptionkey@0"
I20260811 02:37:50.186878 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:50.206832 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:50.210465 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:50.210619 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:50.211265 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:50.211311 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.211337 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:50.211349 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.218945 16085 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:40585
I20260811 02:37:50.219184 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:40585 every 8 connection(s)
I20260811 02:37:50.219473 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415870, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:50.221102 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 16085
I20260811 02:37:50.221386 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:50.221832 3052 external_mini_cluster.cc:1428] Setting key d6b955bed758b260851d93a8ab8b361c
I20260811 02:37:50.224503 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.211929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:46751 (local address 127.2.251.62:39355)
0811 02:37:50.211988 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.211991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.212194 (+ 203us) server_negotiation.cc:407] Connection header received
0811 02:37:50.213224 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.213226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.213242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.213314 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.214565 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.215464 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.217083 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.217315 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.217403 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.220857 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.220860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.220860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.220870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.222490 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.223056 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.223058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.223059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.223136 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.223749 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.223751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.223751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.223893 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.223973 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.224330 (+ 357us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.224428 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 02:37:50.224648 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.211887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.2.251.3:46751)
0811 02:37:50.212069 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.212109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.213051 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.213485 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.213491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.214038 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:50.214442 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.214457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.215575 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.215576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.216980 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.216984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.217432 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.217434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.217632 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.218421 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:50.218432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:50.220592 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:50.222591 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.222602 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.222606 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.222964 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.223296 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.223300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.223300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.223651 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.224001 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:50.224009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:50.224163 (+ 154us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.224271 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.224454 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260811 02:37:50.225242 16316 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39355
I20260811 02:37:50.225350 16316 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:50.225548 16316 heartbeater.cc:511] Master 127.2.251.62:39355 requested a full tablet report, sending...
I20260811 02:37:50.226274 15499 ts_manager.cc:194] Registered new tserver with Master: 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.226812 15499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:46751
I20260811 02:37:50.234911 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:37:50.235316 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.227163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.2.251.3:33575)
0811 02:37:50.227374 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.227387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.227471 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.227749 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.227750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.228030 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:50.228289 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.228297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.229235 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.229236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.230007 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.230011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.230507 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.230509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.230594 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.231225 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:50.231230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:50.231903 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:50.233931 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.233932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.233933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.234226 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.234473 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.234474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.234474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.234569 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.234974 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:50.234975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:50.235040 (+ 65us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.235109 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.235236 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
I20260811 02:37:50.235335 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.227235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:33575 (local address 127.2.251.62:39355)
0811 02:37:50.227290 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.227293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.227484 (+ 191us) server_negotiation.cc:407] Connection header received
0811 02:37:50.227520 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.227521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.227621 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.227719 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.228513 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.229054 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.230140 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.230359 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.230444 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.232040 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.232042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.232043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.232051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.233841 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.234341 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.234342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.234342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.234401 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.234743 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.234744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.234745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.234910 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.234981 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.235148 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.235243 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 02:37:50.262775 15609 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:50.255340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.2.251.62:39355)
0811 02:37:50.255484 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.255487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.255505 (+ 18us) server_negotiation.cc:407] Connection header received
0811 02:37:50.255547 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.255549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.255565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.255669 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.256559 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.257085 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.257956 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.258150 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.258217 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.262650 (+ 4433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33666: BlockingRecv error: recv got EOF from 127.0.0.1:33666 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415870, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:50.271341 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.263768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.2.251.62:39355)
0811 02:37:50.263862 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.263864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.263879 (+ 15us) server_negotiation.cc:407] Connection header received
0811 02:37:50.263954 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.263955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.263966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.264025 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.264585 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.265082 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.266093 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.266320 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.266391 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.268889 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.268891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.268891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.268899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.270300 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.270682 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.270682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.270683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.270729 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.270945 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.270945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.270946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.271050 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.271124 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.271205 (+ 81us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.271270 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 02:37:50.273864 15499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:50.275905 15499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:50.283587 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.278891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:39291 (local address 127.0.0.1:53996)
0811 02:37:50.279738 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.279751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.279825 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.280116 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.280117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.280120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.280177 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.280178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.281316 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.281317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.282288 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.282291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.283216 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.283217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.283305 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.283324 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.283405 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.283484 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":31,"thread_start_us":176,"threads_started":1}
I20260811 02:37:50.283746 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.278782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:40585 (local address 127.0.0.1:49654)
0811 02:37:50.279180 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.279217 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.279293 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.279533 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.279536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.279538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.279615 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.279618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.280937 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.280939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.282472 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.282476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.283377 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.283378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.283535 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.283541 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.283611 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.283660 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":72,"threads_started":1}
I20260811 02:37:50.283883 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.278891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.0.0.1:48052)
0811 02:37:50.279396 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.279407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.279445 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.280198 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.280199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.280202 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.280284 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.280286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.281553 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.281554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.282672 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.282674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.283615 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.283616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.283674 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.283677 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.283725 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.283764 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":4,"thread_start_us":202,"threads_started":1}
I20260811 02:37:50.283972 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.279473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.2.251.2:39291)
0811 02:37:50.279743 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.279748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.279878 (+ 130us) server_negotiation.cc:407] Connection header received
0811 02:37:50.279924 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.279926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.279943 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.280030 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.280252 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.281215 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.282374 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.283114 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.283198 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.283286 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.283826 (+ 540us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.283872 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":131,"threads_started":1}
I20260811 02:37:50.283980 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.279215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.2.251.1:37057)
0811 02:37:50.279961 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.280003 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.280016 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:37:50.280070 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.280072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.280091 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.280189 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.280349 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.281392 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.282763 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.283436 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.283499 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.283577 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.283814 (+ 237us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.283861 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":115,"threads_started":1}
I20260811 02:37:50.284713 15729 tablet_service.cc:1467] Processing CreateTablet for tablet 4110d3a923b749348f39928ed49983c9 (DEFAULT_TABLE table=test-table [id=eaaf06b539ec42e6a55cafc084db9965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.284739 15962 tablet_service.cc:1467] Processing CreateTablet for tablet 4110d3a923b749348f39928ed49983c9 (DEFAULT_TABLE table=test-table [id=eaaf06b539ec42e6a55cafc084db9965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.285096 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4110d3a923b749348f39928ed49983c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.285096 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4110d3a923b749348f39928ed49983c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.288554 16365 tablet_bootstrap.cc:492] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Bootstrap starting.
I20260811 02:37:50.288975 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.279064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.2.251.3:40585)
0811 02:37:50.279356 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.279361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.279372 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:50.279416 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.279420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.279446 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.279531 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.279700 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.280829 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.282586 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.283309 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.283372 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.283527 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.283634 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.288855 (+ 5221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":162,"threads_started":1}
I20260811 02:37:50.289379 16365 tablet_bootstrap.cc:654] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.289778 16365 log.cc:824] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:50.290030 16364 tablet_bootstrap.cc:492] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Bootstrap starting.
I20260811 02:37:50.290053 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 4110d3a923b749348f39928ed49983c9 (DEFAULT_TABLE table=test-table [id=eaaf06b539ec42e6a55cafc084db9965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.290477 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4110d3a923b749348f39928ed49983c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.290925 16364 tablet_bootstrap.cc:654] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.291078 16365 tablet_bootstrap.cc:492] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: No bootstrap required, opened a new log
I20260811 02:37:50.291188 16365 ts_tablet_manager.cc:1397] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:37:50.291373 16364 log.cc:824] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:50.292081 16364 tablet_bootstrap.cc:492] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: No bootstrap required, opened a new log
I20260811 02:37:50.292145 16364 ts_tablet_manager.cc:1397] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:50.293812 16364 raft_consensus.cc:348] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.293773 16365 raft_consensus.cc:348] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.293973 16364 raft_consensus.cc:374] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.293996 16364 raft_consensus.cc:729] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.294101 16364 consensus_queue.cc:260] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.294246 16365 raft_consensus.cc:374] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.294339 16365 raft_consensus.cc:729] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac9525e7df2436a8f4fd4ea2af9359b, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.294389 16364 ts_tablet_manager.cc:1428] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:50.294446 15843 heartbeater.cc:503] Master 127.2.251.62:39355 was elected leader, sending a full tablet report...
I20260811 02:37:50.294570 16365 consensus_queue.cc:260] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.294872 16365 ts_tablet_manager.cc:1428] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 02:37:50.295014 16071 heartbeater.cc:503] Master 127.2.251.62:39355 was elected leader, sending a full tablet report...
I20260811 02:37:50.295367 16370 tablet_bootstrap.cc:492] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Bootstrap starting.
I20260811 02:37:50.296113 16370 tablet_bootstrap.cc:654] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.296422 16370 log.cc:824] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:50.297142 16370 tablet_bootstrap.cc:492] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: No bootstrap required, opened a new log
I20260811 02:37:50.297204 16370 ts_tablet_manager.cc:1397] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:50.298775 16370 raft_consensus.cc:348] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.298943 16370 raft_consensus.cc:374] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.298967 16370 raft_consensus.cc:729] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2be9273ed44348d6aa3b52d1a0604, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.299072 16370 consensus_queue.cc:260] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.299337 16370 ts_tablet_manager.cc:1428] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:37:50.299422 16316 heartbeater.cc:503] Master 127.2.251.62:39355 was elected leader, sending a full tablet report...
W20260811 02:37:50.330411 15846 tablet.cc:2369] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:50.414986 16375 raft_consensus.cc:482] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:50.415143 16375 raft_consensus.cc:504] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.415511 16375 leader_election.cc:288] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057), eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291)
I20260811 02:37:50.419025 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.416087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:36155 (local address 127.2.251.2:39291)
0811 02:37:50.416154 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.416157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.416244 (+ 87us) server_negotiation.cc:407] Connection header received
0811 02:37:50.416342 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.416343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.416358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.416438 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.416667 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.417192 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.418313 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.418694 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.418758 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.418769 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.418912 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.418949 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 02:37:50.419025 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.416087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:53573 (local address 127.2.251.1:37057)
0811 02:37:50.416154 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.416156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.416461 (+ 305us) server_negotiation.cc:407] Connection header received
0811 02:37:50.416499 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.416500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.416515 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.416588 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.416751 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.417214 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.418309 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.418690 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.418751 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.418764 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.418907 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.418953 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 02:37:50.419073 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.415944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.2.251.3:53573)
0811 02:37:50.416349 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.416361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.416432 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.416603 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.416603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.416605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.416684 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.416686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.417296 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.417297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.418223 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.418226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.418770 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.418771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.418842 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.418846 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.418892 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.418948 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":169,"thread_start_us":176,"threads_started":1}
I20260811 02:37:50.419090 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.415944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:39291 (local address 127.2.251.3:36155)
0811 02:37:50.416117 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.416136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.416246 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.416500 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.416501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.416504 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.416591 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.416594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.417287 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.417288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.418222 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.418225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.418773 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.418774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.418844 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.418847 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.418894 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.418940 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":23}
I20260811 02:37:50.419588 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4110d3a923b749348f39928ed49983c9" candidate_uuid: "54f2be9273ed44348d6aa3b52d1a0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" is_pre_election: true
I20260811 02:37:50.419767 15752 raft_consensus.cc:2454] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54f2be9273ed44348d6aa3b52d1a0604 in term 0.
I20260811 02:37:50.419848 15991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4110d3a923b749348f39928ed49983c9" candidate_uuid: "54f2be9273ed44348d6aa3b52d1a0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" is_pre_election: true
I20260811 02:37:50.419997 15991 raft_consensus.cc:2454] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54f2be9273ed44348d6aa3b52d1a0604 in term 0.
I20260811 02:37:50.420044 16112 leader_election.cc:302] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f2be9273ed44348d6aa3b52d1a0604, 6b6a94b43ef342d7be97d55be87dc2b6; no voters:
I20260811 02:37:50.420188 16375 raft_consensus.cc:2781] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:50.420248 16375 raft_consensus.cc:482] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:50.420264 16375 raft_consensus.cc:3037] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.421299 16375 raft_consensus.cc:504] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.421447 16375 leader_election.cc:288] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [CANDIDATE]: Term 1 election: Requested vote from peers 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057), eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291)
I20260811 02:37:50.421627 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4110d3a923b749348f39928ed49983c9" candidate_uuid: "54f2be9273ed44348d6aa3b52d1a0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6a94b43ef342d7be97d55be87dc2b6"
I20260811 02:37:50.421746 15991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4110d3a923b749348f39928ed49983c9" candidate_uuid: "54f2be9273ed44348d6aa3b52d1a0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
I20260811 02:37:50.421794 15752 raft_consensus.cc:3037] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.421844 15991 raft_consensus.cc:3037] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.422727 15991 raft_consensus.cc:2454] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f2be9273ed44348d6aa3b52d1a0604 in term 1.
I20260811 02:37:50.422750 15752 raft_consensus.cc:2454] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f2be9273ed44348d6aa3b52d1a0604 in term 1.
I20260811 02:37:50.423075 16112 leader_election.cc:302] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f2be9273ed44348d6aa3b52d1a0604, 6b6a94b43ef342d7be97d55be87dc2b6; no voters:
I20260811 02:37:50.423190 16375 raft_consensus.cc:2781] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:50.423348 16375 raft_consensus.cc:686] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 LEADER]: Becoming Leader. State: Replica: 54f2be9273ed44348d6aa3b52d1a0604, State: Running, Role: LEADER
I20260811 02:37:50.423436 16375 consensus_queue.cc:237] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } }
I20260811 02:37:50.424577 15499 catalog_manager.cc:5495] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3). New cstate: current_term: 1 leader_uuid: "54f2be9273ed44348d6aa3b52d1a0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:50.431315 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.428956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.2.251.3:40585)
0811 02:37:50.429033 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.429035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.429042 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:37:50.429072 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.429073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.429093 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.429157 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:50.429305 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.429832 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.430576 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.430776 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.430831 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.430991 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.431101 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.431170 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.431209 (+ 39us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.431232 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 02:37:50.433986 15991 raft_consensus.cc:1261] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Refusing update from remote peer 54f2be9273ed44348d6aa3b52d1a0604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.433987 15752 raft_consensus.cc:1261] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Refusing update from remote peer 54f2be9273ed44348d6aa3b52d1a0604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.434455 16379 consensus_queue.cc:1035] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.434629 16375 consensus_queue.cc:1035] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.437249 16388 mvcc.cc:205] Tried to move back new op lower bound from 7317159405293785088 to 7317159405255163904. Current Snapshot: MvccSnapshot[applied={T|T < 7317159405293785088}]
I20260811 02:37:50.438269 16391 mvcc.cc:205] Tried to move back new op lower bound from 7317159405293785088 to 7317159405255163904. Current Snapshot: MvccSnapshot[applied={T|T < 7317159405293785088}]
I20260811 02:37:50.440436 15496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:50.440656 15496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:50.441884 15496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Sending DeleteTablet for 3 replicas of tablet 4110d3a923b749348f39928ed49983c9
I20260811 02:37:50.442306 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet 4110d3a923b749348f39928ed49983c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:48052
I20260811 02:37:50.442374 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 4110d3a923b749348f39928ed49983c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49654
I20260811 02:37:50.442582 16403 tablet_replica.cc:330] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: stopping tablet replica
I20260811 02:37:50.442714 16404 tablet_replica.cc:330] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: stopping tablet replica
I20260811 02:37:50.442727 16403 raft_consensus.cc:2229] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:50.442674 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet 4110d3a923b749348f39928ed49983c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:53996
I20260811 02:37:50.442822 16404 raft_consensus.cc:2229] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:50.442844 16403 raft_consensus.cc:2258] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.442937 16404 raft_consensus.cc:2258] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.443022 15496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:50.443210 15496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:50.443359 16404 ts_tablet_manager.cc:1905] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.443683 16403 ts_tablet_manager.cc:1905] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.445669 16192 tablet_service.cc:1467] Processing CreateTablet for tablet f410255340e44b33b06c1476b90ecf60 (DEFAULT_TABLE table=test-table [id=8dd71cc13a1441858533501819b99e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.445889 15729 tablet_service.cc:1467] Processing CreateTablet for tablet f410255340e44b33b06c1476b90ecf60 (DEFAULT_TABLE table=test-table [id=8dd71cc13a1441858533501819b99e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.446048 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f410255340e44b33b06c1476b90ecf60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.446336 16403 ts_tablet_manager.cc:1918] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:50.446401 16403 log.cc:1196] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/4110d3a923b749348f39928ed49983c9
I20260811 02:37:50.446659 16403 ts_tablet_manager.cc:1939] T 4110d3a923b749348f39928ed49983c9 P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting consensus metadata
I20260811 02:37:50.447153 15476 catalog_manager.cc:4841] TS 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057): tablet 4110d3a923b749348f39928ed49983c9 (table test-table [id=eaaf06b539ec42e6a55cafc084db9965]) successfully deleted
I20260811 02:37:50.447254 16364 tablet_bootstrap.cc:492] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Bootstrap starting.
I20260811 02:37:50.447567 16404 ts_tablet_manager.cc:1918] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:50.447669 16404 log.cc:1196] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/4110d3a923b749348f39928ed49983c9
I20260811 02:37:50.447727 16364 tablet_bootstrap.cc:654] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.447902 16404 ts_tablet_manager.cc:1939] T 4110d3a923b749348f39928ed49983c9 P 54f2be9273ed44348d6aa3b52d1a0604: Deleting consensus metadata
I20260811 02:37:50.448130 16405 tablet_replica.cc:330] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: stopping tablet replica
I20260811 02:37:50.448295 16405 raft_consensus.cc:2229] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:50.448302 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f410255340e44b33b06c1476b90ecf60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.449005 16405 raft_consensus.cc:2258] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.449931 16405 ts_tablet_manager.cc:1905] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.450093 16370 tablet_bootstrap.cc:492] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Bootstrap starting.
I20260811 02:37:50.450553 16370 tablet_bootstrap.cc:654] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.451045 15480 catalog_manager.cc:4841] TS 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585): tablet 4110d3a923b749348f39928ed49983c9 (table test-table [id=eaaf06b539ec42e6a55cafc084db9965]) successfully deleted
I20260811 02:37:50.451494 15962 tablet_service.cc:1467] Processing CreateTablet for tablet f410255340e44b33b06c1476b90ecf60 (DEFAULT_TABLE table=test-table [id=8dd71cc13a1441858533501819b99e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.451635 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f410255340e44b33b06c1476b90ecf60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.452334 16405 ts_tablet_manager.cc:1918] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:50.452399 16405 log.cc:1196] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/4110d3a923b749348f39928ed49983c9
I20260811 02:37:50.452585 16405 ts_tablet_manager.cc:1939] T 4110d3a923b749348f39928ed49983c9 P eac9525e7df2436a8f4fd4ea2af9359b: Deleting consensus metadata
I20260811 02:37:50.453148 16365 tablet_bootstrap.cc:492] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Bootstrap starting.
I20260811 02:37:50.453562 16365 tablet_bootstrap.cc:654] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.453634 15481 catalog_manager.cc:4841] TS eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291): tablet 4110d3a923b749348f39928ed49983c9 (table test-table [id=eaaf06b539ec42e6a55cafc084db9965]) successfully deleted
I20260811 02:37:50.453941 16364 tablet_bootstrap.cc:492] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: No bootstrap required, opened a new log
I20260811 02:37:50.454006 16364 ts_tablet_manager.cc:1397] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260811 02:37:50.454161 16364 raft_consensus.cc:348] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.454241 16364 raft_consensus.cc:374] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.454249 16365 tablet_bootstrap.cc:492] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: No bootstrap required, opened a new log
I20260811 02:37:50.454259 16364 raft_consensus.cc:729] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.454293 16365 ts_tablet_manager.cc:1397] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.454304 16364 consensus_queue.cc:260] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.454402 16364 ts_tablet_manager.cc:1428] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.454438 16365 raft_consensus.cc:348] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.454509 16365 raft_consensus.cc:374] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.454527 16365 raft_consensus.cc:729] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac9525e7df2436a8f4fd4ea2af9359b, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.454573 16365 consensus_queue.cc:260] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.454654 16365 ts_tablet_manager.cc:1428] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:50.454914 16370 tablet_bootstrap.cc:492] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: No bootstrap required, opened a new log
I20260811 02:37:50.454977 16370 ts_tablet_manager.cc:1397] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260811 02:37:50.455133 16370 raft_consensus.cc:348] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.455228 16370 raft_consensus.cc:374] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.455250 16370 raft_consensus.cc:729] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2be9273ed44348d6aa3b52d1a0604, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.455292 16370 consensus_queue.cc:260] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.455380 16370 ts_tablet_manager.cc:1428] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260811 02:37:50.470130 16317 tablet.cc:2369] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:37:50.520884 16368 raft_consensus.cc:482] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:50.521438 16368 raft_consensus.cc:504] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.521878 16368 leader_election.cc:288] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.525667 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.522166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:36947 (local address 127.2.251.2:39291)
0811 02:37:50.522220 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.522222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.522423 (+ 201us) server_negotiation.cc:407] Connection header received
0811 02:37:50.522491 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.522492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.522503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.522553 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.522744 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.523489 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.524741 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.525203 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.525270 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.525283 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.525541 (+ 258us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.525590 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 02:37:50.525866 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.522267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:53011 (local address 127.2.251.3:40585)
0811 02:37:50.522313 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.522315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.522436 (+ 121us) server_negotiation.cc:407] Connection header received
0811 02:37:50.522561 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.522562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.522575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.522633 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:50.522799 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.523282 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.524746 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.525359 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.525446 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.525461 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.525703 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.525788 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:37:50.526206 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.522113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:40585 (local address 127.2.251.1:53011)
0811 02:37:50.522369 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.522379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.522463 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.522646 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.522647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.522648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.522702 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.522703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.523368 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.523369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.524616 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.524620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.525507 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.525509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.525598 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.525604 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.526044 (+ 440us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.526104 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260811 02:37:50.526790 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f410255340e44b33b06c1476b90ecf60" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604" is_pre_election: true
I20260811 02:37:50.526968 16223 raft_consensus.cc:2454] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 0.
I20260811 02:37:50.527025 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.522108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:39291 (local address 127.2.251.1:36947)
0811 02:37:50.522325 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.522339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.522425 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.522566 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.522568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.522570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:50.522656 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.522661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.523585 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.523586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.524638 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.524641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.525350 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.525351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.525446 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.525452 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.526143 (+ 691us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.526202 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":218,"threads_started":1}
I20260811 02:37:50.527194 15653 leader_election.cc:302] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f2be9273ed44348d6aa3b52d1a0604, 6b6a94b43ef342d7be97d55be87dc2b6; no voters:
I20260811 02:37:50.527330 16368 raft_consensus.cc:2781] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:50.527295 15991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f410255340e44b33b06c1476b90ecf60" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" is_pre_election: true
I20260811 02:37:50.527357 16368 raft_consensus.cc:482] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:50.527374 16368 raft_consensus.cc:3037] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.527379 15991 raft_consensus.cc:2454] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 0.
I20260811 02:37:50.528468 16368 raft_consensus.cc:504] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.528615 16368 leader_election.cc:288] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 election: Requested vote from peers eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.528914 15991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f410255340e44b33b06c1476b90ecf60" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
I20260811 02:37:50.528939 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f410255340e44b33b06c1476b90ecf60" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604"
I20260811 02:37:50.529057 16223 raft_consensus.cc:3037] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.529055 15991 raft_consensus.cc:3037] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.529786 16223 raft_consensus.cc:2454] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 1.
I20260811 02:37:50.529945 15653 leader_election.cc:302] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f2be9273ed44348d6aa3b52d1a0604, 6b6a94b43ef342d7be97d55be87dc2b6; no voters:
I20260811 02:37:50.530009 15991 raft_consensus.cc:2454] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 1.
I20260811 02:37:50.530059 16368 raft_consensus.cc:2781] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:50.530221 16368 raft_consensus.cc:686] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Running, Role: LEADER
I20260811 02:37:50.530305 16368 consensus_queue.cc:237] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.531175 15496 catalog_manager.cc:5495] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } health_report { overall_health: UNKNOWN } } }
W20260811 02:37:50.535952 16072 tablet.cc:2369] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415869, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.62@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:50.571588 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.562663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.2.251.62:39355)
0811 02:37:50.562720 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.562722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.562879 (+ 157us) server_negotiation.cc:407] Connection header received
0811 02:37:50.562947 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.562948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.562959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.563014 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.563685 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.564201 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.565398 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.565597 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.565669 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.568807 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.568808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.568808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.568817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.570309 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.570687 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.570688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.570688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.570736 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.570901 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.570902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.570902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.571106 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.571195 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.571360 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.571451 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:37:50.571676 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.562596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39355 (local address 127.0.0.1:33694)
0811 02:37:50.562765 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.562777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.562874 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.563035 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.563036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.563535 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:50.563603 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.563606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.564287 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.564288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.565114 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.565118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.565706 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.565709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.565877 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.566669 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:50.566675 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:50.568708 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:50.570403 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.570413 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.570416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.570616 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.570789 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:50.570790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:50.570790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:50.570848 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:50.571195 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:50.571205 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:50.571258 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.571323 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.571525 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":67,"thread_start_us":82,"threads_started":1}
I20260811 02:37:50.573830 15496 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:33694:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 02:37:50.574054 15496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:50.575826 16192 tablet_service.cc:1467] Processing CreateTablet for tablet fc270ae7dd974695a2db217ff3373b19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2fc292ded8a4dda9424760f9111acef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:50.575826 15729 tablet_service.cc:1467] Processing CreateTablet for tablet fc270ae7dd974695a2db217ff3373b19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2fc292ded8a4dda9424760f9111acef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:50.576032 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc270ae7dd974695a2db217ff3373b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.576028 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc270ae7dd974695a2db217ff3373b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.577116 15962 tablet_service.cc:1467] Processing CreateTablet for tablet fc270ae7dd974695a2db217ff3373b19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b2fc292ded8a4dda9424760f9111acef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 02:37:50.577282 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc270ae7dd974695a2db217ff3373b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.577884 16370 tablet_bootstrap.cc:492] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: Bootstrap starting.
I20260811 02:37:50.578250 16370 tablet_bootstrap.cc:654] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.579131 16365 tablet_bootstrap.cc:492] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: Bootstrap starting.
I20260811 02:37:50.579272 16370 tablet_bootstrap.cc:492] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: No bootstrap required, opened a new log
I20260811 02:37:50.579337 16370 ts_tablet_manager.cc:1397] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.579483 16370 raft_consensus.cc:348] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.579566 16370 raft_consensus.cc:374] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.579587 16370 raft_consensus.cc:729] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2be9273ed44348d6aa3b52d1a0604, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.579617 16365 tablet_bootstrap.cc:654] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.579640 16370 consensus_queue.cc:260] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.579748 16370 ts_tablet_manager.cc:1428] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:50.579890 16398 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.579933 16364 tablet_bootstrap.cc:492] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: Bootstrap starting.
I20260811 02:37:50.579969 16398 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.580415 16364 tablet_bootstrap.cc:654] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.580430 16365 tablet_bootstrap.cc:492] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: No bootstrap required, opened a new log
I20260811 02:37:50.580473 16365 ts_tablet_manager.cc:1397] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.580724 16365 raft_consensus.cc:348] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.580821 16365 raft_consensus.cc:374] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.580842 16365 raft_consensus.cc:729] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac9525e7df2436a8f4fd4ea2af9359b, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.580883 16365 consensus_queue.cc:260] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.581017 16365 ts_tablet_manager.cc:1428] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.581167 16364 tablet_bootstrap.cc:492] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: No bootstrap required, opened a new log
I20260811 02:37:50.581223 16364 ts_tablet_manager.cc:1397] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.581192 16371 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.581262 16371 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.581367 16364 raft_consensus.cc:348] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.581467 16364 raft_consensus.cc:374] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.581498 16364 raft_consensus.cc:729] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.581547 16364 consensus_queue.cc:260] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.581646 16364 ts_tablet_manager.cc:1428] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.581707 16368 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.581804 16368 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.667016 16368 raft_consensus.cc:482] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:50.667148 16368 raft_consensus.cc:504] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.667338 16368 leader_election.cc:288] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.667707 15991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc270ae7dd974695a2db217ff3373b19" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" is_pre_election: true
I20260811 02:37:50.667668 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc270ae7dd974695a2db217ff3373b19" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604" is_pre_election: true
I20260811 02:37:50.667801 15991 raft_consensus.cc:2454] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 0.
I20260811 02:37:50.667809 16223 raft_consensus.cc:2454] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 0.
I20260811 02:37:50.667968 15654 leader_election.cc:302] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6a94b43ef342d7be97d55be87dc2b6, eac9525e7df2436a8f4fd4ea2af9359b; no voters:
I20260811 02:37:50.668097 16368 raft_consensus.cc:2781] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:50.668165 16368 raft_consensus.cc:482] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:50.668184 16368 raft_consensus.cc:3037] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.669064 16368 raft_consensus.cc:504] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.669204 16368 leader_election.cc:288] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 election: Requested vote from peers eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.669464 15991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc270ae7dd974695a2db217ff3373b19" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
I20260811 02:37:50.669461 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc270ae7dd974695a2db217ff3373b19" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604"
I20260811 02:37:50.669550 15991 raft_consensus.cc:3037] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.669553 16223 raft_consensus.cc:3037] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.670264 15991 raft_consensus.cc:2454] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 1.
I20260811 02:37:50.670289 16223 raft_consensus.cc:2454] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 1.
I20260811 02:37:50.670455 15654 leader_election.cc:302] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6a94b43ef342d7be97d55be87dc2b6, eac9525e7df2436a8f4fd4ea2af9359b; no voters:
I20260811 02:37:50.670575 16368 raft_consensus.cc:2781] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:50.670631 16368 raft_consensus.cc:686] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Running, Role: LEADER
I20260811 02:37:50.670691 16368 consensus_queue.cc:237] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.671110 16413 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: TxnStatusTablet state changed. Reason: New leader 6b6a94b43ef342d7be97d55be87dc2b6. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.671211 16413 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:50.671434 16442 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 02:37:50.671564 16442 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 02:37:50.671517 15499 catalog_manager.cc:5495] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:50.678771 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.675524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.2.251.1:37057)
0811 02:37:50.675587 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.675589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.675599 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:50.675633 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.675635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.675655 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.675739 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:50.675849 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.676607 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.677516 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.677879 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.677958 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.678330 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.678481 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.678573 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.678656 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.678684 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 02:37:50.678802 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.675356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.0.0.1:48062)
0811 02:37:50.675444 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.675459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.675525 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.675740 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.675741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.675742 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:50.675802 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.675803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.676688 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.676689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.677430 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.677434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.677980 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.677981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.678047 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.678068 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.678558 (+ 490us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.678564 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.678644 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.678707 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 02:37:50.680377 16223 raft_consensus.cc:1261] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Refusing update from remote peer 6b6a94b43ef342d7be97d55be87dc2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.680375 15991 raft_consensus.cc:1261] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Refusing update from remote peer 6b6a94b43ef342d7be97d55be87dc2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.680687 16413 consensus_queue.cc:1035] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.680855 16368 consensus_queue.cc:1035] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.681566 16398 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: TxnStatusTablet state changed. Reason: New leader 6b6a94b43ef342d7be97d55be87dc2b6. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.681671 16398 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.681646 16371 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: TxnStatusTablet state changed. Reason: New leader 6b6a94b43ef342d7be97d55be87dc2b6. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.681789 16371 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.682480 16394 mvcc.cc:205] Tried to move back new op lower bound from 7317159406304088064 to 7317159406267613184. Current Snapshot: MvccSnapshot[applied={T|T < 7317159406304088064}]
I20260811 02:37:50.682600 16413 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.682677 16413 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:50.682730 16413 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.682766 16413 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 6b6a94b43ef342d7be97d55be87dc2b6: This TxnStatusTablet replica's current role is: LEADER
I20260811 02:37:50.682972 16371 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.683060 16371 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P eac9525e7df2436a8f4fd4ea2af9359b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.683068 16398 tablet_replica.cc:439] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } } }
I20260811 02:37:50.683161 16398 tablet_replica.cc:442] T fc270ae7dd974695a2db217ff3373b19 P 54f2be9273ed44348d6aa3b52d1a0604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 02:37:50.687299 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.684682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.2.251.1:37057)
0811 02:37:50.684717 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.684719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.684727 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:50.684772 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.684773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.684784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.684826 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:50.685056 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.685541 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.686670 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.686892 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.686958 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.687094 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.687123 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.687151 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.687220 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.687236 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260811 02:37:50.694121 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.690798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:56609 (local address 127.2.251.1:37057)
0811 02:37:50.690863 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.690865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.690972 (+ 107us) server_negotiation.cc:407] Connection header received
0811 02:37:50.691045 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.691046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.691055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.691104 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:50.691340 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.692085 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.693295 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.693491 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.693553 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.693827 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.693870 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.693925 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.693993 (+ 68us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.694029 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":43}
I20260811 02:37:50.694233 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.690745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.2.251.1:56609)
0811 02:37:50.690915 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:50.690926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:50.690989 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:50.691110 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:50.691111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:50.691121 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:50.691192 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.691194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.692181 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.692181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.693184 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:50.693187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.693593 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.693594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:50.693658 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.693683 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.693937 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.693938 (+ 1us) client_negotiation.cc:769] Sending connection context
0811 02:37:50.693989 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 02:37:50.694056 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260811 02:37:50.696964 16223 raft_consensus.cc:1261] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Refusing update from remote peer 6b6a94b43ef342d7be97d55be87dc2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.697445 15991 raft_consensus.cc:1261] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Refusing update from remote peer 6b6a94b43ef342d7be97d55be87dc2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.697767 16368 consensus_queue.cc:1035] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.697927 16413 consensus_queue.cc:1035] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.716936 15494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:50.717094 15494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:50.718185 15494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Sending DeleteTablet for 3 replicas of tablet f410255340e44b33b06c1476b90ecf60
I20260811 02:37:50.718676 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet f410255340e44b33b06c1476b90ecf60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49654
I20260811 02:37:50.718824 16404 tablet_replica.cc:330] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: stopping tablet replica
I20260811 02:37:50.718813 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet f410255340e44b33b06c1476b90ecf60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:53996
I20260811 02:37:50.718916 16404 raft_consensus.cc:2229] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:50.718977 16404 raft_consensus.cc:2258] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.719017 16405 tablet_replica.cc:330] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: stopping tablet replica
I20260811 02:37:50.719038 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet f410255340e44b33b06c1476b90ecf60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:48052
I20260811 02:37:50.719090 16405 raft_consensus.cc:2229] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:50.719137 16405 raft_consensus.cc:2258] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.719187 16403 tablet_replica.cc:330] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: stopping tablet replica
I20260811 02:37:50.719254 16403 raft_consensus.cc:2229] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:50.719365 16403 raft_consensus.cc:2258] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.719713 16404 ts_tablet_manager.cc:1905] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.719702 16405 ts_tablet_manager.cc:1905] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.721509 16403 ts_tablet_manager.cc:1905] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.722445 16405 ts_tablet_manager.cc:1918] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:50.722522 16405 log.cc:1196] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/f410255340e44b33b06c1476b90ecf60
I20260811 02:37:50.722750 16405 ts_tablet_manager.cc:1939] T f410255340e44b33b06c1476b90ecf60 P eac9525e7df2436a8f4fd4ea2af9359b: Deleting consensus metadata
I20260811 02:37:50.723105 16404 ts_tablet_manager.cc:1918] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:50.723166 16404 log.cc:1196] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/f410255340e44b33b06c1476b90ecf60
I20260811 02:37:50.723539 16404 ts_tablet_manager.cc:1939] T f410255340e44b33b06c1476b90ecf60 P 54f2be9273ed44348d6aa3b52d1a0604: Deleting consensus metadata
I20260811 02:37:50.723662 15481 catalog_manager.cc:4841] TS eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291): tablet f410255340e44b33b06c1476b90ecf60 (table test-table [id=8dd71cc13a1441858533501819b99e0e]) successfully deleted
I20260811 02:37:50.723887 16403 ts_tablet_manager.cc:1918] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:50.723949 16403 log.cc:1196] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/f410255340e44b33b06c1476b90ecf60
I20260811 02:37:50.724181 16403 ts_tablet_manager.cc:1939] T f410255340e44b33b06c1476b90ecf60 P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting consensus metadata
I20260811 02:37:50.724241 15480 catalog_manager.cc:4841] TS 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585): tablet f410255340e44b33b06c1476b90ecf60 (table test-table [id=8dd71cc13a1441858533501819b99e0e]) successfully deleted
I20260811 02:37:50.724527 15476 catalog_manager.cc:4841] TS 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057): tablet f410255340e44b33b06c1476b90ecf60 (table test-table [id=8dd71cc13a1441858533501819b99e0e]) successfully deleted
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 02:37:50.752223 15609 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:50.745564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.2.251.62:39355)
0811 02:37:50.745647 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.745649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.745707 (+ 58us) server_negotiation.cc:407] Connection header received
0811 02:37:50.745814 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.745815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.745829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.745912 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.746539 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.747306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.748333 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.748652 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.748738 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.752146 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33700: BlockingRecv error: recv got EOF from 127.0.0.1:33700 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 11 02:37:50 dist-test-slave-cvwc krb5kdc[15429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM
I20260811 02:37:50.763211 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.753285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.2.251.62:39355)
0811 02:37:50.753328 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.753330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.753370 (+ 40us) server_negotiation.cc:407] Connection header received
0811 02:37:50.753446 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.753446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.753457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.753505 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:50.754308 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.754902 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.756140 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.756362 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.756444 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.759497 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:50.759499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:50.759499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:50.759509 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:50.761526 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.762133 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.762135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.762135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.762218 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:50.762453 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:50.762454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:50.762454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:50.762631 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:50.762763 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.763006 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.763108 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:37:50.765569 15494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:50.766018 15494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:50.768090 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb487ea14bf4ab186304d9e9714568c (DEFAULT_TABLE table=test-table [id=d48575c49ac04e58b35a851348efdb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.768095 15729 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb487ea14bf4ab186304d9e9714568c (DEFAULT_TABLE table=test-table [id=d48575c49ac04e58b35a851348efdb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.768277 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb487ea14bf4ab186304d9e9714568c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.768105 15962 tablet_service.cc:1467] Processing CreateTablet for tablet 7eb487ea14bf4ab186304d9e9714568c (DEFAULT_TABLE table=test-table [id=d48575c49ac04e58b35a851348efdb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.768419 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb487ea14bf4ab186304d9e9714568c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.768420 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eb487ea14bf4ab186304d9e9714568c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.769824 16370 tablet_bootstrap.cc:492] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Bootstrap starting.
I20260811 02:37:50.769972 16364 tablet_bootstrap.cc:492] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Bootstrap starting.
I20260811 02:37:50.770255 16370 tablet_bootstrap.cc:654] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.770316 16364 tablet_bootstrap.cc:654] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.770641 16365 tablet_bootstrap.cc:492] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Bootstrap starting.
I20260811 02:37:50.770972 16370 tablet_bootstrap.cc:492] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: No bootstrap required, opened a new log
I20260811 02:37:50.771025 16370 ts_tablet_manager.cc:1397] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:50.771076 16365 tablet_bootstrap.cc:654] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.771155 16370 raft_consensus.cc:348] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.771237 16370 raft_consensus.cc:374] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.771256 16370 raft_consensus.cc:729] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2be9273ed44348d6aa3b52d1a0604, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.771293 16370 consensus_queue.cc:260] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.771386 16370 ts_tablet_manager.cc:1428] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.772073 16365 tablet_bootstrap.cc:492] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: No bootstrap required, opened a new log
I20260811 02:37:50.772132 16365 ts_tablet_manager.cc:1397] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:50.772297 16365 raft_consensus.cc:348] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.772382 16365 raft_consensus.cc:374] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.772401 16365 raft_consensus.cc:729] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac9525e7df2436a8f4fd4ea2af9359b, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.772452 16365 consensus_queue.cc:260] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.772550 16365 ts_tablet_manager.cc:1428] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:50.773133 16364 tablet_bootstrap.cc:492] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: No bootstrap required, opened a new log
I20260811 02:37:50.773198 16364 ts_tablet_manager.cc:1397] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 02:37:50.773386 16364 raft_consensus.cc:348] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.773458 16364 raft_consensus.cc:374] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.773475 16364 raft_consensus.cc:729] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.773521 16364 consensus_queue.cc:260] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.773617 16364 ts_tablet_manager.cc:1428] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:50.835069 16368 raft_consensus.cc:482] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:50.835203 16368 raft_consensus.cc:504] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.835388 16368 leader_election.cc:288] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.835641 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb487ea14bf4ab186304d9e9714568c" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604" is_pre_election: true
I20260811 02:37:50.835677 15989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb487ea14bf4ab186304d9e9714568c" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" is_pre_election: true
I20260811 02:37:50.835765 16223 raft_consensus.cc:2454] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 0.
I20260811 02:37:50.835775 15989 raft_consensus.cc:2454] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 0.
I20260811 02:37:50.835950 15654 leader_election.cc:302] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6a94b43ef342d7be97d55be87dc2b6, eac9525e7df2436a8f4fd4ea2af9359b; no voters:
I20260811 02:37:50.836053 16368 raft_consensus.cc:2781] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:50.836107 16368 raft_consensus.cc:482] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:50.836133 16368 raft_consensus.cc:3037] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.837116 16368 raft_consensus.cc:504] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.837287 16368 leader_election.cc:288] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 election: Requested vote from peers eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:50.837509 15989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb487ea14bf4ab186304d9e9714568c" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac9525e7df2436a8f4fd4ea2af9359b"
I20260811 02:37:50.837592 15989 raft_consensus.cc:3037] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.838346 15989 raft_consensus.cc:2454] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 1.
I20260811 02:37:50.838500 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eb487ea14bf4ab186304d9e9714568c" candidate_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604"
I20260811 02:37:50.838613 16223 raft_consensus.cc:3037] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:50.838647 15654 leader_election.cc:302] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6a94b43ef342d7be97d55be87dc2b6, eac9525e7df2436a8f4fd4ea2af9359b; no voters:
I20260811 02:37:50.838755 16368 raft_consensus.cc:2781] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:50.838835 16368 raft_consensus.cc:686] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Running, Role: LEADER
I20260811 02:37:50.838893 16368 consensus_queue.cc:237] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.839479 16223 raft_consensus.cc:2454] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6a94b43ef342d7be97d55be87dc2b6 in term 1.
I20260811 02:37:50.839564 15494 catalog_manager.cc:5495] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:50.852883 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:50.849737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.2.251.1:37057)
0811 02:37:50.849780 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:50.849781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:50.849790 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:50.849833 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:50.849834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:50.849845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:50.849891 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:50.850122 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.850853 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.852015 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:50.852284 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:50.852348 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:50.852546 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:50.852584 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:50.852624 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:50.852712 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 02:37:50.852771 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:37:50.855005 15989 raft_consensus.cc:1261] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Refusing update from remote peer 6b6a94b43ef342d7be97d55be87dc2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.855010 16223 raft_consensus.cc:1261] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Refusing update from remote peer 6b6a94b43ef342d7be97d55be87dc2b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:50.855232 16413 consensus_queue.cc:1035] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.855356 16448 consensus_queue.cc:1035] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:50.858328 15494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:50.858424 15494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:50.859129 15494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Sending DeleteTablet for 3 replicas of tablet 7eb487ea14bf4ab186304d9e9714568c
I20260811 02:37:50.859526 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb487ea14bf4ab186304d9e9714568c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49654
I20260811 02:37:50.859539 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb487ea14bf4ab186304d9e9714568c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:53996
I20260811 02:37:50.859627 16404 tablet_replica.cc:330] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: stopping tablet replica
I20260811 02:37:50.859649 16405 tablet_replica.cc:330] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: stopping tablet replica
I20260811 02:37:50.859675 16404 raft_consensus.cc:2229] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:50.859651 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eb487ea14bf4ab186304d9e9714568c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:50 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:48052
I20260811 02:37:50.859704 16405 raft_consensus.cc:2229] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:50.859715 16404 raft_consensus.cc:2258] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.859757 16405 raft_consensus.cc:2258] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.859751 16403 tablet_replica.cc:330] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: stopping tablet replica
I20260811 02:37:50.859797 16403 raft_consensus.cc:2229] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:50.859861 16403 raft_consensus.cc:2258] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:50.860051 16405 ts_tablet_manager.cc:1905] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.860342 15494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:37:50.860499 15494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:37:50.861853 16403 ts_tablet_manager.cc:1905] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.861835 15962 tablet_service.cc:1467] Processing CreateTablet for tablet f0ba215a04c241fda9804a46f080667f (DEFAULT_TABLE table=test-table [id=20b3997055634afabb0c00308ee4d6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.862083 15729 tablet_service.cc:1467] Processing CreateTablet for tablet f0ba215a04c241fda9804a46f080667f (DEFAULT_TABLE table=test-table [id=20b3997055634afabb0c00308ee4d6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.862216 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ba215a04c241fda9804a46f080667f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.862843 16405 ts_tablet_manager.cc:1918] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:50.862916 16405 log.cc:1196] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/7eb487ea14bf4ab186304d9e9714568c
I20260811 02:37:50.863142 16405 ts_tablet_manager.cc:1939] T 7eb487ea14bf4ab186304d9e9714568c P eac9525e7df2436a8f4fd4ea2af9359b: Deleting consensus metadata
I20260811 02:37:50.863478 16403 ts_tablet_manager.cc:1918] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:50.863540 16364 tablet_bootstrap.cc:492] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6: Bootstrap starting.
I20260811 02:37:50.863620 16192 tablet_service.cc:1467] Processing CreateTablet for tablet f0ba215a04c241fda9804a46f080667f (DEFAULT_TABLE table=test-table [id=20b3997055634afabb0c00308ee4d6c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:37:50.863672 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ba215a04c241fda9804a46f080667f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.863742 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ba215a04c241fda9804a46f080667f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:50.863879 16364 tablet_bootstrap.cc:654] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.865017 16365 tablet_bootstrap.cc:492] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Bootstrap starting.
I20260811 02:37:50.865446 15481 catalog_manager.cc:4841] TS eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291): tablet 7eb487ea14bf4ab186304d9e9714568c (table test-table [id=d48575c49ac04e58b35a851348efdb7d]) successfully deleted
I20260811 02:37:50.865680 16365 tablet_bootstrap.cc:654] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.865820 16370 tablet_bootstrap.cc:492] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Bootstrap starting.
I20260811 02:37:50.865870 16364 tablet_bootstrap.cc:492] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6: No bootstrap required, opened a new log
I20260811 02:37:50.865916 16364 ts_tablet_manager.cc:1397] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:50.866051 16364 raft_consensus.cc:348] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.866132 16364 raft_consensus.cc:374] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.866155 16364 raft_consensus.cc:729] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6a94b43ef342d7be97d55be87dc2b6, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.866158 16370 tablet_bootstrap.cc:654] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:50.866221 16364 consensus_queue.cc:260] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.866305 16365 tablet_bootstrap.cc:492] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: No bootstrap required, opened a new log
I20260811 02:37:50.866333 16364 ts_tablet_manager.cc:1428] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:50.866348 16365 ts_tablet_manager.cc:1397] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:50.866503 16365 raft_consensus.cc:348] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.866566 16365 raft_consensus.cc:374] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.866595 16365 raft_consensus.cc:729] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac9525e7df2436a8f4fd4ea2af9359b, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.863541 16403 log.cc:1196] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/7eb487ea14bf4ab186304d9e9714568c
I20260811 02:37:50.866636 16365 consensus_queue.cc:260] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.866712 16365 ts_tablet_manager.cc:1428] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260811 02:37:50.866863 16403 ts_tablet_manager.cc:1939] T 7eb487ea14bf4ab186304d9e9714568c P 6b6a94b43ef342d7be97d55be87dc2b6: Deleting consensus metadata
I20260811 02:37:50.867249 15476 catalog_manager.cc:4841] TS 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057): tablet 7eb487ea14bf4ab186304d9e9714568c (table test-table [id=d48575c49ac04e58b35a851348efdb7d]) successfully deleted
I20260811 02:37:50.867571 16404 ts_tablet_manager.cc:1905] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:50.867946 16370 tablet_bootstrap.cc:492] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: No bootstrap required, opened a new log
I20260811 02:37:50.867993 16370 ts_tablet_manager.cc:1397] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:50.868104 16370 raft_consensus.cc:348] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.868176 16370 raft_consensus.cc:374] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:50.868197 16370 raft_consensus.cc:729] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f2be9273ed44348d6aa3b52d1a0604, State: Initialized, Role: FOLLOWER
I20260811 02:37:50.868263 16370 consensus_queue.cc:260] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:50.868378 16370 ts_tablet_manager.cc:1428] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:37:50.869458 16404 ts_tablet_manager.cc:1918] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:37:50.869637 16404 log.cc:1196] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/7eb487ea14bf4ab186304d9e9714568c
I20260811 02:37:50.869977 16404 ts_tablet_manager.cc:1939] T 7eb487ea14bf4ab186304d9e9714568c P 54f2be9273ed44348d6aa3b52d1a0604: Deleting consensus metadata
I20260811 02:37:50.870344 15480 catalog_manager.cc:4841] TS 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585): tablet 7eb487ea14bf4ab186304d9e9714568c (table test-table [id=d48575c49ac04e58b35a851348efdb7d]) successfully deleted
I20260811 02:37:51.086527 16371 raft_consensus.cc:482] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:37:51.086658 16371 raft_consensus.cc:504] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:51.087309 16371 leader_election.cc:288] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:51.090267 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.087543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.2.251.2:33011)
0811 02:37:51.087676 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:51.087688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:51.087797 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:51.087941 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:51.087942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:51.087944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:51.088009 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.088012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.088646 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.088647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.089405 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.089407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.090016 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.090017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.090077 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.090081 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:51.090128 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:37:51.090175 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260811 02:37:51.090272 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.087622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:33011 (local address 127.2.251.1:37057)
0811 02:37:51.087687 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:51.087690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:51.087765 (+ 75us) server_negotiation.cc:407] Connection header received
0811 02:37:51.087871 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:51.087872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:51.087881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:51.087927 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:51.088068 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.088544 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.089506 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.089920 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.089989 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.090000 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:51.090142 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 02:37:51.090175 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 02:37:51.090588 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ba215a04c241fda9804a46f080667f" candidate_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" is_pre_election: true
I20260811 02:37:51.090694 15752 raft_consensus.cc:2454] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac9525e7df2436a8f4fd4ea2af9359b in term 0.
I20260811 02:37:51.090831 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.087543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:40585 (local address 127.2.251.2:39013)
0811 02:37:51.087711 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:51.087723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:51.087803 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:51.087990 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:51.087991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:51.087993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:37:51.088058 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.088059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.089053 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.089055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.089910 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.089913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.090575 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.090576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.090660 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.090664 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:37:51.090712 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 02:37:51.090751 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260811 02:37:51.090912 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.087626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:39013 (local address 127.2.251.3:40585)
0811 02:37:51.087772 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:51.087774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:51.087785 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:51.087897 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:51.087898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:51.087909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:51.087981 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:37:51.088196 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.088945 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.089986 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.090491 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.090570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.090585 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:51.090742 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 02:37:51.090796 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260811 02:37:51.090898 15881 leader_election.cc:302] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6a94b43ef342d7be97d55be87dc2b6, eac9525e7df2436a8f4fd4ea2af9359b; no voters:
I20260811 02:37:51.091053 16371 raft_consensus.cc:2781] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:37:51.091085 16371 raft_consensus.cc:482] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:37:51.091102 16371 raft_consensus.cc:3037] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:51.091112 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ba215a04c241fda9804a46f080667f" candidate_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604" is_pre_election: true
I20260811 02:37:51.091223 16223 raft_consensus.cc:2454] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac9525e7df2436a8f4fd4ea2af9359b in term 0.
I20260811 02:37:51.091919 16371 raft_consensus.cc:504] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:51.092125 16371 leader_election.cc:288] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [CANDIDATE]: Term 1 election: Requested vote from peers 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057), 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585)
I20260811 02:37:51.092482 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ba215a04c241fda9804a46f080667f" candidate_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f2be9273ed44348d6aa3b52d1a0604"
I20260811 02:37:51.092563 16223 raft_consensus.cc:3037] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:51.092487 15752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ba215a04c241fda9804a46f080667f" candidate_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6a94b43ef342d7be97d55be87dc2b6"
I20260811 02:37:51.092705 15752 raft_consensus.cc:3037] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:51.093324 16223 raft_consensus.cc:2454] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac9525e7df2436a8f4fd4ea2af9359b in term 1.
I20260811 02:37:51.093525 15883 leader_election.cc:302] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f2be9273ed44348d6aa3b52d1a0604, eac9525e7df2436a8f4fd4ea2af9359b; no voters:
I20260811 02:37:51.093600 15752 raft_consensus.cc:2454] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac9525e7df2436a8f4fd4ea2af9359b in term 1.
I20260811 02:37:51.093667 16371 raft_consensus.cc:2781] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:51.093855 16371 raft_consensus.cc:686] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 1 LEADER]: Becoming Leader. State: Replica: eac9525e7df2436a8f4fd4ea2af9359b, State: Running, Role: LEADER
I20260811 02:37:51.093938 16371 consensus_queue.cc:237] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } }
I20260811 02:37:51.094841 15494 catalog_manager.cc:5495] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b reported cstate change: term changed from 0 to 1, leader changed from <none> to eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2). New cstate: current_term: 1 leader_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac9525e7df2436a8f4fd4ea2af9359b" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 } health_report { overall_health: UNKNOWN } } }
I20260811 02:37:51.159169 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.155560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.2.251.2:39291)
0811 02:37:51.155803 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:51.155805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:51.155816 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:51.155860 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:51.155860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:51.155883 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:51.155971 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:51.156199 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.156861 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.157762 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.158050 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.158129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.158434 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:51.158583 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:51.158668 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:51.158847 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 02:37:51.158901 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":133,"threads_started":1}
I20260811 02:37:51.168237 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.164127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:39787 (local address 127.2.251.1:37057)
0811 02:37:51.164232 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:51.164234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:51.164248 (+ 14us) server_negotiation.cc:407] Connection header received
0811 02:37:51.164395 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:51.164396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:51.164407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:51.164592 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:51.164906 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.165846 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.167109 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.167344 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.167413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.167741 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:51.167777 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:51.167825 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:51.168085 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 02:37:51.168128 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 02:37:51.168244 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.163882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.2.251.2:39787)
0811 02:37:51.164198 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:51.164211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:51.164267 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:51.164621 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:51.164622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:51.164624 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:51.164742 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.164745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.166001 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.166003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.166955 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.166962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.167547 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.167549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.167612 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.167636 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:51.167928 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:51.167934 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:37:51.168022 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 02:37:51.168094 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":149,"threads_started":1}
I20260811 02:37:51.171244 16223 raft_consensus.cc:1261] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Refusing update from remote peer eac9525e7df2436a8f4fd4ea2af9359b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:51.171294 15752 raft_consensus.cc:1261] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Refusing update from remote peer eac9525e7df2436a8f4fd4ea2af9359b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:37:51.171875 16501 consensus_queue.cc:1035] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f2be9273ed44348d6aa3b52d1a0604" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:51.172056 16371 consensus_queue.cc:1035] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b6a94b43ef342d7be97d55be87dc2b6" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:37:51.181577 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.178653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:35523 (local address 127.2.251.2:39291)
0811 02:37:51.178706 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:51.178708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:51.178718 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:51.178817 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:51.178818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:51.178834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:51.178918 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:37:51.179074 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.179761 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.180601 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.180837 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.180922 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.181237 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:37:51.181285 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:37:51.181357 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:51.181432 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 02:37:51.181471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:37:51.181612 16451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.178592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:39291 (local address 127.2.251.1:35523)
0811 02:37:51.178669 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:51.178682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:51.178729 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:51.178918 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:51.178919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:51.178921 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 02:37:51.178992 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.179002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.179850 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.179852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.180450 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:51.180452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:51.180970 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:51.180971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:51.181058 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:51.181069 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 02:37:51.181364 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 02:37:51.181367 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:37:51.181423 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 02:37:51.181502 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260811 02:37:51.194639 15494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:51.194803 15494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:37:51.195833 15494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f11a5203ee9499988df4ff4d2cd5f7c: Sending DeleteTablet for 3 replicas of tablet f0ba215a04c241fda9804a46f080667f
I20260811 02:37:51.196151 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ba215a04c241fda9804a46f080667f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:51 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:53996
I20260811 02:37:51.196316 16192 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ba215a04c241fda9804a46f080667f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:51 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49654
I20260811 02:37:51.196427 16405 tablet_replica.cc:330] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: stopping tablet replica
I20260811 02:37:51.196485 16405 raft_consensus.cc:2229] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:37:51.196484 15729 tablet_service.cc:1514] Processing DeleteTablet for tablet f0ba215a04c241fda9804a46f080667f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:37:51 UTC) from {username='oryx', principal='oryx/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:48052
I20260811 02:37:51.196578 16403 tablet_replica.cc:330] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6: stopping tablet replica
I20260811 02:37:51.196589 16405 raft_consensus.cc:2258] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:51.196632 16403 raft_consensus.cc:2229] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:51.196673 16403 raft_consensus.cc:2258] T f0ba215a04c241fda9804a46f080667f P 6b6a94b43ef342d7be97d55be87dc2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:51.196666 16404 tablet_replica.cc:330] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: stopping tablet replica
I20260811 02:37:51.196738 16404 raft_consensus.cc:2229] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:37:51.196719 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 15624
I20260811 02:37:51.196791 16404 raft_consensus.cc:2258] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:37:51.221745 16405 ts_tablet_manager.cc:1905] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:51.223735 16405 ts_tablet_manager.cc:1918] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:51.223824 16405 log.cc:1196] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/f0ba215a04c241fda9804a46f080667f
I20260811 02:37:51.224027 16405 ts_tablet_manager.cc:1939] T f0ba215a04c241fda9804a46f080667f P eac9525e7df2436a8f4fd4ea2af9359b: Deleting consensus metadata
I20260811 02:37:51.224466 15481 catalog_manager.cc:4841] TS eac9525e7df2436a8f4fd4ea2af9359b (127.2.251.2:39291): tablet f0ba215a04c241fda9804a46f080667f (table test-table [id=20b3997055634afabb0c00308ee4d6c6]) successfully deleted
I20260811 02:37:51.226166 16404 ts_tablet_manager.cc:1905] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:37:51.227474 16404 ts_tablet_manager.cc:1918] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 02:37:51.227533 16404 log.cc:1196] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/f0ba215a04c241fda9804a46f080667f
I20260811 02:37:51.227718 16404 ts_tablet_manager.cc:1939] T f0ba215a04c241fda9804a46f080667f P 54f2be9273ed44348d6aa3b52d1a0604: Deleting consensus metadata
I20260811 02:37:51.228127 15480 catalog_manager.cc:4841] TS 54f2be9273ed44348d6aa3b52d1a0604 (127.2.251.3:40585): tablet f0ba215a04c241fda9804a46f080667f (table test-table [id=20b3997055634afabb0c00308ee4d6c6]) successfully deleted
W20260811 02:37:51.234979 15476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:37057 (error 108)
I20260811 02:37:51.235443 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 15856
I20260811 02:37:51.245388 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:51.240113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:37057 (local address 127.0.0.1:48092)
0811 02:37:51.245191 (+ 5078us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:51.245315 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:37057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5047,"thread_start_us":67,"threads_started":1}
W20260811 02:37:51.245597 15476 catalog_manager.cc:4584] TS 6b6a94b43ef342d7be97d55be87dc2b6 (127.2.251.1:37057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0ba215a04c241fda9804a46f080667f: Network error: Client connection negotiation failed: client connection to 127.2.251.1:37057: connect: Connection refused (error 111)
I20260811 02:37:51.250406 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 16085
I20260811 02:37:51.285364 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 15451
2026-08-11T02:37:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3259 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:51 dist-test-slave-cvwc krb5kdc[16535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:51 dist-test-slave-cvwc krb5kdc[16535](info): set up 2 sockets
Aug 11 02:37:51 dist-test-slave-cvwc krb5kdc[16535](info): commencing operation
krb5kdc: starting...
W20260811 02:37:52.562716 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:52 dist-test-slave-cvwc krb5kdc[16535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:52.701601 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34183
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:52.796350 16557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:52.796593 16557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:52.796618 16557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:52.797926 16557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:52.797978 16557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:52.797998 16557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:52.798017 16557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:52.798031 16557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:52.800329 16557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:52.801558 16557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:52.802031 16557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:52.808739 16565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:52.808779 16570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:52.808902 16557 server_base.cc:1034] running on GCE node
W20260811 02:37:52.808781 16566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:52.809273 16557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:52.809603 16557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:52.810763 16557 hybrid_clock.cc:648] HybridClock initialized: now 1786415872810740 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T02:37:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1514 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:37:52 dist-test-slave-cvwc krb5kdc[16579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:37:52 dist-test-slave-cvwc krb5kdc[16579](info): set up 2 sockets
Aug 11 02:37:52 dist-test-slave-cvwc krb5kdc[16579](info): commencing operation
krb5kdc: starting...
W20260811 02:37:54.051594 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:37:54.189534 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:35649
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:54.280071 16597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:54.280359 16597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:54.280400 16597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:54.281801 16597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:54.281862 16597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:54.281884 16597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:54.281904 16597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:54.281931 16597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:54.283677 16597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:54.283985 16597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:54.284233 16597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:54.290539 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.290551 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.290552 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:54.290913 16597 server_base.cc:1034] running on GCE node
I20260811 02:37:54.291186 16597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:54.291474 16597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:54.292650 16597 hybrid_clock.cc:648] HybridClock initialized: now 1786415874292637 us; error 31 us; skew 500 ppm
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:54.295532 16597 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:54.296079 16597 webserver.cc:460] Webserver started at http://127.2.251.62:41653/ using document root <none> and password file <none>
I20260811 02:37:54.296314 16597 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:54.296365 16597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:54.296492 16597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:54.297683 16597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "67d00133a9054119a5b8807b152cfb16"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "d301862bc8457b74bf9e6e95191caf5c"
server_key_iv: "2bb221a2c42f1cc5808866918bc5434f"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.298151 16597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "67d00133a9054119a5b8807b152cfb16"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "d301862bc8457b74bf9e6e95191caf5c"
server_key_iv: "2bb221a2c42f1cc5808866918bc5434f"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.299924 16597 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260811 02:37:54.300937 16623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.301499 16597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:54.301622 16597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "67d00133a9054119a5b8807b152cfb16"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "d301862bc8457b74bf9e6e95191caf5c"
server_key_iv: "2bb221a2c42f1cc5808866918bc5434f"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.301700 16597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:54.325650 16597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:54.328276 16597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:54.328418 16597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:54.332909 16597 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:35649
I20260811 02:37:54.332959 16721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:35649 every 8 connection(s)
I20260811 02:37:54.333351 16597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:54.334354 16723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:37:54.336248 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 16597
I20260811 02:37:54.336349 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:37:54.336565 3052 external_mini_cluster.cc:1428] Setting key f92bac01e26f515e95b444bf33368576
I20260811 02:37:54.336970 16723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Bootstrap starting.
I20260811 02:37:54.337690 16723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Neither blocks nor log segments found. Creating new log.
I20260811 02:37:54.338004 16723 log.cc:824] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:54.338825 16723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: No bootstrap required, opened a new log
I20260811 02:37:54.340951 16723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:54.341120 16723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:37:54.341143 16723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d00133a9054119a5b8807b152cfb16, State: Initialized, Role: FOLLOWER
I20260811 02:37:54.341261 16723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:54.341343 16723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:54.341383 16723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:54.341430 16723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:37:54.342196 16723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:54.342322 16723 leader_election.cc:302] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d00133a9054119a5b8807b152cfb16; no voters:
I20260811 02:37:54.342526 16723 leader_election.cc:288] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:37:54.342610 16729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:37:54.342818 16729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 LEADER]: Becoming Leader. State: Replica: 67d00133a9054119a5b8807b152cfb16, State: Running, Role: LEADER
I20260811 02:37:54.342816 16723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:54.342991 16729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:54.344266 16731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d00133a9054119a5b8807b152cfb16. Latest consensus state: current_term: 1 leader_uuid: "67d00133a9054119a5b8807b152cfb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } } }
I20260811 02:37:54.344354 16731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:54.344595 16754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:54.344878 16730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67d00133a9054119a5b8807b152cfb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } } }
I20260811 02:37:54.345016 16730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:54.345140 16754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:54.347805 16754 catalog_manager.cc:1347] Generated new cluster ID: 13d0e043f78c4f0688746e6558978043
I20260811 02:37:54.347868 16754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:54.348738 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.337134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.2.251.62:35649)
0811 02:37:54.337302 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:54.337309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:54.337338 (+ 29us) server_negotiation.cc:407] Connection header received
0811 02:37:54.337849 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:54.337859 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:54.337919 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:54.338055 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:54.338881 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.339828 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.340914 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.341469 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.341566 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.344799 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:54.344804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:54.344804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:54.344818 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:54.346915 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.347416 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.347419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.347420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.347537 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.347770 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.347770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.347771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.348100 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:54.348251 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:54.348382 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 02:37:54.348530 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260811 02:37:54.354521 16754 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:37:54.355228 16754 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:54.363759 16754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Generated new TSK 0
I20260811 02:37:54.364012 16754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:37:54.397190 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35649
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:54.491565 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:54.491838 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:54.491868 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:54.493185 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:54.493257 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:54.493291 16764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:54.495018 16764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:54.495337 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:54.495596 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:37:54.501623 16764 server_base.cc:1034] running on GCE node
W20260811 02:37:54.501590 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.501745 16778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.501565 16779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:54.502667 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:54.502957 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:54.504139 16764 hybrid_clock.cc:648] HybridClock initialized: now 1786415874504050 us; error 108 us; skew 500 ppm
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:54.507628 16764 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:37:54.508316 16764 webserver.cc:460] Webserver started at http://127.2.251.1:42125/ using document root <none> and password file <none>
I20260811 02:37:54.508570 16764 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:54.508628 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:54.508778 16764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:54.510041 16764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "8b3264f1bfbd4d10854c4e7a02ba5f50"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "92a6d1ce3ffde2d4c490e9e9d6b4f646"
server_key_iv: "341aa67317109babaccf559f73ceb10c"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.510440 16764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "8b3264f1bfbd4d10854c4e7a02ba5f50"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "92a6d1ce3ffde2d4c490e9e9d6b4f646"
server_key_iv: "341aa67317109babaccf559f73ceb10c"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.511951 16764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:54.512828 16796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.513002 16764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:54.513104 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "8b3264f1bfbd4d10854c4e7a02ba5f50"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "92a6d1ce3ffde2d4c490e9e9d6b4f646"
server_key_iv: "341aa67317109babaccf559f73ceb10c"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.513171 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:54.527037 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:54.530090 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:54.530258 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:54.530508 16764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:54.530869 16764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:54.530915 16764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.530954 16764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:54.530979 16764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.537416 16764 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:43499
I20260811 02:37:54.537540 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:43499 every 8 connection(s)
I20260811 02:37:54.537916 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:54.545317 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 16764
I20260811 02:37:54.545449 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:37:54.545658 3052 external_mini_cluster.cc:1428] Setting key b88cfbe415d7c8feeebac3c3fc9edc6c
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:54.551028 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.538768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:57093 (local address 127.2.251.62:35649)
0811 02:37:54.538839 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:54.538842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:54.539100 (+ 258us) server_negotiation.cc:407] Connection header received
0811 02:37:54.539588 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:54.539589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:54.539603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:54.539664 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:54.540802 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.541458 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.542767 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.543119 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.543197 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.546963 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:54.546967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:54.546967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:54.546978 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:54.549108 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.549847 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.549850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.549850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.549948 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.550277 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.550277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.550278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.550458 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:54.550532 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:54.550810 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 02:37:54.550931 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:37:54.551105 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.538725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:57093)
0811 02:37:54.538932 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:54.538974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:54.539489 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:54.539824 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:54.539828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:54.540449 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:54.540658 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:54.540663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.541578 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.541579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:54.542618 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:54.542622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.543269 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.543273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:54.543831 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.544680 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:54.544693 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:54.546803 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:54.549272 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:54.549286 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:54.549290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:54.549686 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:54.550029 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:54.550030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:54.550031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:54.550201 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:54.550520 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:54.550524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:54.550672 (+ 148us) client_negotiation.cc:769] Sending connection context
0811 02:37:54.550756 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 02:37:54.550907 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260811 02:37:54.551915 16993 heartbeater.cc:347] Connected to a master server at 127.2.251.62:35649
I20260811 02:37:54.552059 16993 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:54.552268 16993 heartbeater.cc:511] Master 127.2.251.62:35649 requested a full tablet report, sending...
I20260811 02:37:54.552887 16655 ts_manager.cc:194] Registered new tserver with Master: 8b3264f1bfbd4d10854c4e7a02ba5f50 (127.2.251.1:43499)
I20260811 02:37:54.554095 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:57093
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:37:54.582247 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35649
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:54.673341 17000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:54.673583 17000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:54.673611 17000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:54.674968 17000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:54.675034 17000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:54.675065 17000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:54.676669 17000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:54.676934 17000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:54.677181 17000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:54.683322 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:54.683560 17000 server_base.cc:1034] running on GCE node
W20260811 02:37:54.683323 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.683322 17014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:54.683956 17000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:54.684234 17000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:54.685417 17000 hybrid_clock.cc:648] HybridClock initialized: now 1786415874685414 us; error 58 us; skew 500 ppm
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:54.689113 17000 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:37:54.689690 17000 webserver.cc:460] Webserver started at http://127.2.251.2:44271/ using document root <none> and password file <none>
I20260811 02:37:54.689990 17000 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:54.690048 17000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:54.690198 17000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:54.691390 17000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "2f09aa704e224847a4f7cfa2f544bd87"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "eb56d776af89c32f8e92f861602c4631"
server_key_iv: "fcf7684d98f2daf988da834bdbbf4c63"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.691783 17000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "2f09aa704e224847a4f7cfa2f544bd87"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "eb56d776af89c32f8e92f861602c4631"
server_key_iv: "fcf7684d98f2daf988da834bdbbf4c63"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.693439 17000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:37:54.694478 17022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.694674 17000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:54.694790 17000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "2f09aa704e224847a4f7cfa2f544bd87"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "eb56d776af89c32f8e92f861602c4631"
server_key_iv: "fcf7684d98f2daf988da834bdbbf4c63"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.694864 17000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:54.711881 17000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:54.714293 17000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:54.714452 17000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:54.714713 17000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:54.715037 17000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:54.715071 17000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.715101 17000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:54.715130 17000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.722393 17000 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:39605
I20260811 02:37:54.722491 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:39605 every 8 connection(s)
I20260811 02:37:54.723038 17000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:54.729655 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 17000
I20260811 02:37:54.729802 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:37:54.730005 3052 external_mini_cluster.cc:1428] Setting key c17cfd5c85a3e905a4b8d24b4a066c1b
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:54.734951 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.724058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:51937 (local address 127.2.251.62:35649)
0811 02:37:54.724245 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:54.724247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:54.724396 (+ 149us) server_negotiation.cc:407] Connection header received
0811 02:37:54.725672 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:54.725674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:54.725692 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:54.725827 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:54.726553 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.727164 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.728353 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.728562 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.728622 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.731460 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:54.731462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:54.731462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:54.731471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:54.733154 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.733892 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.733893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.733894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.733957 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.734246 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.734247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.734247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.734375 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:54.734442 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:54.734736 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 02:37:54.734838 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260811 02:37:54.735054 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.723971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:51937)
0811 02:37:54.724248 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:54.724306 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:54.725428 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:54.725915 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:54.725921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:54.726230 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:54.726448 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:54.726453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.727293 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.727295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:54.728248 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:54.728251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.728638 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.728639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:54.728768 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.729310 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:54.729319 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:54.731315 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:54.733288 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:54.733303 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:54.733307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:54.733739 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:54.734021 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:54.734022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:54.734022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:54.734180 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:54.734439 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:54.734445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:54.734598 (+ 153us) client_negotiation.cc:769] Sending connection context
0811 02:37:54.734694 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 02:37:54.734853 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":122,"threads_started":1}
I20260811 02:37:54.735610 17227 heartbeater.cc:347] Connected to a master server at 127.2.251.62:35649
I20260811 02:37:54.735731 17227 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:54.735965 17227 heartbeater.cc:511] Master 127.2.251.62:35649 requested a full tablet report, sending...
I20260811 02:37:54.736532 16655 ts_manager.cc:194] Registered new tserver with Master: 2f09aa704e224847a4f7cfa2f544bd87 (127.2.251.2:39605)
I20260811 02:37:54.737196 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:51937
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:37:54.766568 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35649
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:54.865422 17237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:54.865756 17237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:54.865797 17237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:54.867681 17237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:54.867760 17237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:54.867810 17237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:54.870332 17237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:54.870662 17237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:54.870983 17237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:54.881884 17249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.881963 17243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:54.882166 17245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:54.882439 17237 server_base.cc:1034] running on GCE node
I20260811 02:37:54.882663 17237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:54.882915 17237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:54.884100 17237 hybrid_clock.cc:648] HybridClock initialized: now 1786415874884057 us; error 37 us; skew 500 ppm
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:54.887549 17237 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:37:54.890506 17237 webserver.cc:460] Webserver started at http://127.2.251.3:43281/ using document root <none> and password file <none>
I20260811 02:37:54.890738 17237 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:54.890805 17237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:54.890954 17237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:37:54.892496 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "8d2437216b004437bf490999767c4cd1"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "02ee6c98f3bc7c7e9b7c00967c4aa68a"
server_key_iv: "430e1d9500c1b8fede20e0b6757638c6"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.892992 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "8d2437216b004437bf490999767c4cd1"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "02ee6c98f3bc7c7e9b7c00967c4aa68a"
server_key_iv: "430e1d9500c1b8fede20e0b6757638c6"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.894976 17237 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 02:37:54.896019 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.896304 17237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:54.896420 17237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "8d2437216b004437bf490999767c4cd1"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "02ee6c98f3bc7c7e9b7c00967c4aa68a"
server_key_iv: "430e1d9500c1b8fede20e0b6757638c6"
server_key_version: "encryptionkey@0"
I20260811 02:37:54.896503 17237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:54.918483 17237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:54.921255 17237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:54.921448 17237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:54.921741 17237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:54.922483 17237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:54.922545 17237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.922582 17237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:54.922600 17237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:54.931852 17237 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:37379
I20260811 02:37:54.931964 17451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:37379 every 8 connection(s)
I20260811 02:37:54.932289 17237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:54.935879 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 17237
I20260811 02:37:54.935972 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:37:54.936136 3052 external_mini_cluster.cc:1428] Setting key 28c446b2d9965654b1562abc56608ca0
Aug 11 02:37:54 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:37:54.944442 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.933130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:43615 (local address 127.2.251.62:35649)
0811 02:37:54.933231 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:54.933233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:54.933398 (+ 165us) server_negotiation.cc:407] Connection header received
0811 02:37:54.934010 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:54.934011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:54.934023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:54.934077 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:54.935079 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.935585 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.936765 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.936988 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.937059 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.940348 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:54.940350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:54.940350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:54.940358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:54.942033 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.943127 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.943130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.943131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.943248 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:54.943542 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:54.943543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:54.943543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:54.943718 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:54.943801 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:54.944274 (+ 473us) server_negotiation.cc:299] Negotiation successful
0811 02:37:54.944363 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 02:37:54.944526 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:54.933056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:43615)
0811 02:37:54.933267 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:54.933314 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:54.933910 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:54.934149 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:54.934153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:54.934779 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:54.934987 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:54.934992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.935687 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.935688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:54.936684 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:54.936687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:54.937349 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:54.937352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:54.937695 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:54.938393 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:54.938403 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:54.940209 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:37:54.942140 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:54.942149 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:54.942152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:54.943025 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:54.943339 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:37:54.943339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:37:54.943340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:37:54.943480 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:37:54.943831 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:37:54.943838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:37:54.944056 (+ 218us) client_negotiation.cc:769] Sending connection context
0811 02:37:54.944170 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 02:37:54.944320 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260811 02:37:54.945217 17452 heartbeater.cc:347] Connected to a master server at 127.2.251.62:35649
I20260811 02:37:54.945506 17452 heartbeater.cc:464] Registering TS with master...
I20260811 02:37:54.945758 17452 heartbeater.cc:511] Master 127.2.251.62:35649 requested a full tablet report, sending...
I20260811 02:37:54.946292 16655 ts_manager.cc:194] Registered new tserver with Master: 8d2437216b004437bf490999767c4cd1 (127.2.251.3:37379)
I20260811 02:37:54.946722 16655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:43615
I20260811 02:37:54.948619 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab.
Entry for principal oryx/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab.
WARNING: no policy specified for oryx/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab.
Entry for principal oryx/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab.
WARNING: no policy specified for oryx/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab.
Entry for principal oryx/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab.
I20260811 02:37:55.048751 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 16764
I20260811 02:37:55.097407 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 17000
I20260811 02:37:55.107569 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 17237
I20260811 02:37:55.117904 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 16597
I20260811 02:37:55.130741 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:35649
--webserver_interface=127.2.251.62
--webserver_port=41653
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:37:55.226016 17469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:55.226526 17469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:55.226570 17469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:55.228351 17469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:37:55.228447 17469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:55.228462 17469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:55.228475 17469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:37:55.228488 17469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:37:55.230368 17469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:55.230681 17469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:55.230937 17469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:55.236209 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.236209 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:55.236356 17469 server_base.cc:1034] running on GCE node
W20260811 02:37:55.236235 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:55.236809 17469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:55.237116 17469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:55.238358 17469 hybrid_clock.cc:648] HybridClock initialized: now 1786415875238305 us; error 50 us; skew 500 ppm
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415875, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:55.241791 17469 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:37:55.242354 17469 webserver.cc:460] Webserver started at http://127.2.251.62:41653/ using document root <none> and password file <none>
I20260811 02:37:55.242700 17469 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:55.242784 17469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:55.244655 17469 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:55.245507 17495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.245999 17469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:55.246342 17469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "67d00133a9054119a5b8807b152cfb16"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "d301862bc8457b74bf9e6e95191caf5c"
server_key_iv: "2bb221a2c42f1cc5808866918bc5434f"
server_key_version: "encryptionkey@0"
I20260811 02:37:55.246929 17469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:55.260051 17469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:55.263834 17469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:55.263975 17469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:55.268581 17469 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:35649
I20260811 02:37:55.268636 17591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:35649 every 8 connection(s)
I20260811 02:37:55.269021 17469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:37:55.270673 17594 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:37:55.271389 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Bootstrap starting.
I20260811 02:37:55.277115 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 17469
I20260811 02:37:55.277537 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:43499
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:35649
--webserver_port=42125
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:37:55.281006 17594 log.cc:824] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Log is configured to *not* fsync() on all Append() calls
I20260811 02:37:55.282888 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:37:55.283185 17594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Bootstrap complete.
I20260811 02:37:55.286033 17594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:55.286314 17594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d00133a9054119a5b8807b152cfb16, State: Initialized, Role: FOLLOWER
I20260811 02:37:55.286463 17594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:55.286540 17594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:37:55.286589 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:37:55.286636 17594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 1 FOLLOWER]: Advancing to term 2
I20260811 02:37:55.287756 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:55.287868 17594 leader_election.cc:302] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d00133a9054119a5b8807b152cfb16; no voters:
I20260811 02:37:55.288077 17594 leader_election.cc:288] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 02:37:55.288132 17601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 02:37:55.288255 17601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [term 2 LEADER]: Becoming Leader. State: Replica: 67d00133a9054119a5b8807b152cfb16, State: Running, Role: LEADER
I20260811 02:37:55.288347 17601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } }
I20260811 02:37:55.288375 17594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:37:55.289685 17605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d00133a9054119a5b8807b152cfb16. Latest consensus state: current_term: 2 leader_uuid: "67d00133a9054119a5b8807b152cfb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } } }
I20260811 02:37:55.289687 17602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "67d00133a9054119a5b8807b152cfb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d00133a9054119a5b8807b152cfb16" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35649 } } }
I20260811 02:37:55.290024 17602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:55.290138 17605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16 [sys.catalog]: This master's current role is: LEADER
I20260811 02:37:55.290949 17631 catalog_manager.cc:1259] Loaded cluster ID: 13d0e043f78c4f0688746e6558978043
I20260811 02:37:55.290997 17631 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: loading cluster ID for follower catalog manager: success
I20260811 02:37:55.292145 17631 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: acquiring CA information for follower catalog manager: success
I20260811 02:37:55.292521 17631 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:37:55.292615 17634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:37:55.292997 17634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:37:55.293092 17634 catalog_manager.cc:1259] Loaded cluster ID: 13d0e043f78c4f0688746e6558978043
I20260811 02:37:55.293125 17634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:37:55.293300 17634 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:37:55.293408 17634 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67d00133a9054119a5b8807b152cfb16: Loaded TSK: 0
I20260811 02:37:55.293561 17634 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 02:37:55.380132 17596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:55.380368 17596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:55.380405 17596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:55.381654 17596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:55.381702 17596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:55.381763 17596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:37:55.383328 17596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.1.keytab
--principal=oryx/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:43499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=42125
--tserver_master_addrs=127.2.251.62:35649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:55.383574 17596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:55.383806 17596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:55.389293 17641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.389261 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.389240 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:55.389624 17596 server_base.cc:1034] running on GCE node
I20260811 02:37:55.389914 17596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:55.390192 17596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:55.391327 17596 hybrid_clock.cc:648] HybridClock initialized: now 1786415875391308 us; error 33 us; skew 500 ppm
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415875, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:55.394763 17596 init.cc:377] Logged in from keytab as oryx/127.2.251.1@KRBTEST.COM (short username oryx)
I20260811 02:37:55.395396 17596 webserver.cc:460] Webserver started at http://127.2.251.1:42125/ using document root <none> and password file <none>
I20260811 02:37:55.395690 17596 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:55.395776 17596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:55.397676 17596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:37:55.398432 17657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.398622 17596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:37:55.398728 17596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "8b3264f1bfbd4d10854c4e7a02ba5f50"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "92a6d1ce3ffde2d4c490e9e9d6b4f646"
server_key_iv: "341aa67317109babaccf559f73ceb10c"
server_key_version: "encryptionkey@0"
I20260811 02:37:55.399205 17596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:55.415740 17596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:55.418666 17596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:55.418826 17596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:55.419140 17596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:55.419513 17596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:55.419557 17596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.419602 17596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:55.419625 17596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.431296 17596 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:43499
I20260811 02:37:55.431425 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:43499 every 8 connection(s)
I20260811 02:37:55.431973 17596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:37:55.437134 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 17596
I20260811 02:37:55.437557 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:39605
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:35649
--webserver_port=44271
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.445794 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.433296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:47455)
0811 02:37:55.433587 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.433647 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.434845 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.435680 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.435686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.436685 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.437045 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.437055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.439078 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.439080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.440558 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.440564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.441622 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.441625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.441874 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.442564 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.442577 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.445499 (+ 2922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":136,"threads_started":1}
W20260811 02:37:55.445878 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.433420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:47455 (local address 127.2.251.62:35649)
0811 02:37:55.433653 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.433662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.433777 (+ 115us) server_negotiation.cc:407] Connection header received
0811 02:37:55.435285 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.435293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.435382 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.435581 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.437338 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.438857 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.440717 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.441427 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.441582 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.445672 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:47455: BlockingRecv error: recv got EOF from 127.2.251.1:47455 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
W20260811 02:37:55.446053 17870 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:35649 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.251.62:35649: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.452387 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.446311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:48043)
0811 02:37:55.446366 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.446376 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.446421 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.446580 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.446581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.446737 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.446812 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.446814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.447528 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.447530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.448478 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.448483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.449112 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.449114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.449183 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.449672 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.449678 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.452307 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 02:37:55.452576 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.446350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:48043 (local address 127.2.251.62:35649)
0811 02:37:55.446395 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.446397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.446427 (+ 30us) server_negotiation.cc:407] Connection header received
0811 02:37:55.446489 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.446490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.446502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.446553 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.446920 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.447448 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.448580 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.448872 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.448949 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.452492 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:48043: BlockingRecv error: recv got EOF from 127.2.251.1:48043 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.458483 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.452661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:42217)
0811 02:37:55.452812 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.452821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.452873 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.453010 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.453011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.453155 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.453198 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.453199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.453901 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.453902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.454718 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.454720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.455094 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.455095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.455151 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.455731 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.455736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.458382 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
W20260811 02:37:55.458650 17870 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:37:55.458772 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.452823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:42217 (local address 127.2.251.62:35649)
0811 02:37:55.452895 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.452896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.452904 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:55.452931 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.452931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.452940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.452994 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.453297 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.453828 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.454821 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.455020 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.455092 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.458671 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:42217: BlockingRecv error: recv got EOF from 127.2.251.1:42217 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260811 02:37:55.538574 17875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:55.538849 17875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:55.538893 17875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:55.540330 17875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:55.540388 17875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:55.540431 17875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:37:55.542186 17875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.2.keytab
--principal=oryx/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:39605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=44271
--tserver_master_addrs=127.2.251.62:35649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:55.542485 17875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:55.542744 17875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:55.549291 17906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.549307 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.550138 17903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:55.550246 17875 server_base.cc:1034] running on GCE node
I20260811 02:37:55.550573 17875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:55.550901 17875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:55.552107 17875 hybrid_clock.cc:648] HybridClock initialized: now 1786415875552086 us; error 40 us; skew 500 ppm
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415875, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:55.555076 17875 init.cc:377] Logged in from keytab as oryx/127.2.251.2@KRBTEST.COM (short username oryx)
I20260811 02:37:55.555593 17875 webserver.cc:460] Webserver started at http://127.2.251.2:44271/ using document root <none> and password file <none>
I20260811 02:37:55.555830 17875 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:55.555900 17875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:55.557492 17875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:37:55.558338 17918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.558530 17875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:37:55.558627 17875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "2f09aa704e224847a4f7cfa2f544bd87"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "eb56d776af89c32f8e92f861602c4631"
server_key_iv: "fcf7684d98f2daf988da834bdbbf4c63"
server_key_version: "encryptionkey@0"
I20260811 02:37:55.559173 17875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:55.572353 17875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:55.575096 17875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:55.575233 17875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:55.575467 17875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:55.575795 17875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:55.575829 17875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.575852 17875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:55.575863 17875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.582752 17875 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:39605
I20260811 02:37:55.582808 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:39605 every 8 connection(s)
I20260811 02:37:55.583195 17875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:37:55.585314 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 17875
I20260811 02:37:55.585660 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:37379
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:35649
--webserver_port=43281
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 02:37:55.592792 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.584049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:33695 (local address 127.2.251.62:35649)
0811 02:37:55.584138 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.584141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.584272 (+ 131us) server_negotiation.cc:407] Connection header received
0811 02:37:55.584818 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.584819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.584836 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.584887 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.586364 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.586981 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.588314 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.588635 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.588825 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.592703 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:33695: BlockingRecv error: recv got EOF from 127.2.251.2:33695 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 02:37:55.592799 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.583968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:33695)
0811 02:37:55.584140 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.584182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.584717 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.584958 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.584962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.585856 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.586165 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.586174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.587174 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.587175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.588165 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.588169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.588872 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.588873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.589018 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.589695 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.589707 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.592542 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
W20260811 02:37:55.593075 18127 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:35649 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.251.62:35649: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.598110 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.593330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:51263)
0811 02:37:55.593382 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.593393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.593455 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.593589 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.593590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.593760 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.593829 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.593832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.594499 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.594500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.595041 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.595044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.595384 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.595385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.595449 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.595893 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.595898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.598029 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 02:37:55.598198 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.593391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:51263 (local address 127.2.251.62:35649)
0811 02:37:55.593450 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.593453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.593463 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:55.593519 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.593520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.593531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.593579 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.593899 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.594421 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.595118 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.595321 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.595378 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.598141 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:51263: BlockingRecv error: recv got EOF from 127.2.251.2:51263 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.605676 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.598738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:40509)
0811 02:37:55.598800 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.598811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.598863 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.598987 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.598988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.599123 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.599160 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.599162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.600876 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.600878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.601863 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.601867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.602199 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.602200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.602255 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.602685 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.602690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.605591 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 02:37:55.605862 18127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:37:55.605865 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.598807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:40509 (local address 127.2.251.62:35649)
0811 02:37:55.598863 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.598865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.598875 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:55.598921 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.598922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.598932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.598978 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.599574 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.600636 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.601942 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.602133 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.602190 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.605766 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:40509: BlockingRecv error: recv got EOF from 127.2.251.2:40509 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260811 02:37:55.679632 18132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:37:55.679939 18132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:37:55.680024 18132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:37:55.681768 18132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:37:55.681842 18132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:37:55.681941 18132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:37:55.683856 18132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/krb5kdc/oryx_127.2.251.3.keytab
--principal=oryx/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:37379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=43281
--tserver_master_addrs=127.2.251.62:35649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:37:55.684132 18132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:37:55.684501 18132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:37:55.690891 18146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.691032 18139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:37:55.690899 18141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:37:55.691118 18132 server_base.cc:1034] running on GCE node
I20260811 02:37:55.691360 18132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:37:55.691668 18132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:37:55.692904 18132 hybrid_clock.cc:648] HybridClock initialized: now 1786415875692880 us; error 34 us; skew 500 ppm
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415875, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:37:55.695744 18132 init.cc:377] Logged in from keytab as oryx/127.2.251.3@KRBTEST.COM (short username oryx)
I20260811 02:37:55.696301 18132 webserver.cc:460] Webserver started at http://127.2.251.3:43281/ using document root <none> and password file <none>
I20260811 02:37:55.696547 18132 fs_manager.cc:359] Metadata directory not provided
I20260811 02:37:55.696617 18132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:37:55.698271 18132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:37:55.699193 18157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.699568 18132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:37:55.699693 18132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "8d2437216b004437bf490999767c4cd1"
format_stamp: "Formatted at 2026-08-11 02:37:54 on dist-test-slave-cvwc"
server_key: "02ee6c98f3bc7c7e9b7c00967c4aa68a"
server_key_iv: "430e1d9500c1b8fede20e0b6757638c6"
server_key_version: "encryptionkey@0"
I20260811 02:37:55.700147 18132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:37:55.733234 18132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:37:55.736029 18132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:37:55.736200 18132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:37:55.736482 18132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:37:55.736829 18132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:37:55.736878 18132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.736910 18132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:37:55.736927 18132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:37:55.743549 18132 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:37379
I20260811 02:37:55.743636 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:37379 every 8 connection(s)
I20260811 02:37:55.744050 18132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:37:55.744585 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 18132
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.754746 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.745353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.2.251.62:35649)
0811 02:37:55.745589 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.745591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.745602 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:55.745638 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.745639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.745655 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.745760 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.746697 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.747499 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.748515 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.748752 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.748867 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.750939 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:37:55.750944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:37:55.750945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:37:55.750960 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:37:55.753146 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:55.753586 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:55.753590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:55.753591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:55.753752 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:37:55.753953 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:37:55.753954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:37:55.753955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:37:55.754220 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:37:55.754366 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 02:37:55.754495 (+ 129us) server_negotiation.cc:299] Negotiation successful
0811 02:37:55.754638 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
W20260811 02:37:55.754820 17874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.745022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:42861 (local address 127.2.251.62:35649)
0811 02:37:55.745122 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.745125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.745709 (+ 584us) server_negotiation.cc:407] Connection header received
0811 02:37:55.746427 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.746428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.746440 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.746523 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.747987 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.748643 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.750167 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.750560 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.750673 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.754734 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:42861: BlockingRecv error: recv got EOF from 127.2.251.3:42861 (error 108)
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":52}
I20260811 02:37:55.754848 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.744951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:42861)
0811 02:37:55.745387 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.745539 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.746195 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.746726 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.746733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.747538 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.747787 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.747796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.748770 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.748772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.750023 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.750027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.750687 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.750689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.750814 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.751708 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.751719 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.754605 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":103,"threads_started":1}
W20260811 02:37:55.755108 18345 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:35649 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.251.62:35649: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.760797 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.755492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:33315)
0811 02:37:55.755545 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.755555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.755591 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.755716 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.755716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.755867 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.755928 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.755930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.756650 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.756652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.757410 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.757416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.757974 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.757975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.758052 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.758558 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.758563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.760727 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 02:37:55.760922 18351 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.755533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:33315 (local address 127.2.251.62:35649)
0811 02:37:55.755571 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.755573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.755581 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:55.755636 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.755637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.755648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.755697 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.756006 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.756497 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.757600 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.757879 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.757938 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.760854 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:33315: BlockingRecv error: recv got EOF from 127.2.251.3:33315 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:55 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:55.766717 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:55.761172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:33563)
0811 02:37:55.761227 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:55.761239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:55.761275 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:55.761546 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:55.761547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:55.761670 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:55.761751 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.761754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.762769 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.762771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.763352 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:55.763354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.763761 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.763763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:55.763818 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.764374 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:55.764379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:55.766641 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 02:37:55.766866 18351 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:55.761348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:33563 (local address 127.2.251.62:35649)
0811 02:37:55.761399 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:55.761401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:55.761409 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:37:55.761433 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:55.761433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:55.761441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:55.761487 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:55.761927 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.762563 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.763453 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:55.763671 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:55.763734 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:55.766745 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:33563: BlockingRecv error: recv got EOF from 127.2.251.3:33563 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260811 02:37:55.766896 18345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 02:37:56 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:56 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:56.468377 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:56.459128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:52883)
0811 02:37:56.459292 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:56.459304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:56.459370 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:56.459546 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:56.459547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:56.459738 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:56.459787 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:56.459788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.460510 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.460511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:56.461071 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:56.461073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.461857 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.461858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:56.461912 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:56.465637 (+ 3725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:56.465646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:56.468275 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
W20260811 02:37:56.468636 18356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:56.459265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:52883 (local address 127.2.251.62:35649)
0811 02:37:56.459397 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:56.459399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:56.459408 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:56.459433 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:56.459433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:56.459443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:56.459503 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:56.459878 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.460409 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.461167 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.461620 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.461847 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:56.468401 (+ 6554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:52883: BlockingRecv error: recv got EOF from 127.2.251.1:52883 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
Aug 11 02:37:56 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:56 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:56.613517 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:56.606684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:43775)
0811 02:37:56.606930 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:56.606944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:56.607019 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:56.607235 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:56.607236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:56.607864 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:56.607995 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:56.608001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.609175 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.609177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:56.609877 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:56.609880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.610388 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.610389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:56.610467 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:56.611053 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:56.611060 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:56.613429 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
W20260811 02:37:56.613677 18356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:56.606828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:43775 (local address 127.2.251.62:35649)
0811 02:37:56.606889 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:56.606891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:56.607012 (+ 121us) server_negotiation.cc:407] Connection header received
0811 02:37:56.607076 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:56.607077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:56.607089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:56.607131 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:56.608159 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.608976 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.610027 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.610250 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.610320 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:56.613606 (+ 3286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:43775: BlockingRecv error: recv got EOF from 127.2.251.2:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 11 02:37:56 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:56 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:56.772956 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:56.767403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:51111)
0811 02:37:56.767587 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:56.767599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:56.767665 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:56.767872 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:56.767873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:56.768124 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:56.768209 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:56.768212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.769010 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.769012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:56.769591 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:56.769594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.770053 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.770054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:56.770117 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:56.770640 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:56.770645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:56.772875 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
W20260811 02:37:56.773100 18356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:56.767579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:51111 (local address 127.2.251.62:35649)
0811 02:37:56.767679 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:56.767682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:56.767722 (+ 40us) server_negotiation.cc:407] Connection header received
0811 02:37:56.767759 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:56.767760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:56.767778 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:56.767842 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:56.768386 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.768904 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.769694 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:56.769950 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:56.770023 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:56.773027 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:51111: BlockingRecv error: recv got EOF from 127.2.251.3:51111 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 11 02:37:57 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:57 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:57.474241 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:57.469076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:56249)
0811 02:37:57.469317 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:57.469330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:57.469401 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:57.469771 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:57.469773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:57.469998 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:57.470064 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:57.470067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.470758 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.470759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:57.471311 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:57.471313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.471739 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.471740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:57.471802 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:57.472345 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:57.472350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:57.474174 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":148,"threads_started":1}
W20260811 02:37:57.474432 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:57.469417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:56249 (local address 127.2.251.62:35649)
0811 02:37:57.469596 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:57.469598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:57.469605 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:37:57.469636 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:57.469637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:57.469650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:57.469750 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:57.470147 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.470661 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.471419 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.471629 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.471695 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:57.474348 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:56249: BlockingRecv error: recv got EOF from 127.2.251.1:56249 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
Aug 11 02:37:57 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:57 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:57.620211 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:57.614175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:51611)
0811 02:37:57.614335 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:57.614347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:57.614414 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:57.614592 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:57.614594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:57.614770 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:57.614847 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:57.614849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.615548 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.615549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:57.616135 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:57.616137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.616727 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.616728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:57.616792 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:57.617391 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:57.617396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:57.620107 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
W20260811 02:37:57.620468 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:57.614325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:51611 (local address 127.2.251.62:35649)
0811 02:37:57.614389 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:57.614391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:57.614401 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:57.614478 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:57.614479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:57.614490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:57.614532 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:57.614950 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.615456 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.616317 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.616587 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.616701 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:57.620391 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:51611: BlockingRecv error: recv got EOF from 127.2.251.2:51611 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 11 02:37:57 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:57 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:57.778250 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:57.773520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:60849)
0811 02:37:57.773749 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:57.773768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:57.773837 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:57.773972 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:57.773972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:57.774145 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:57.774198 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:57.774199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.774905 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.774906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:57.775470 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:57.775472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.775863 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.775865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:57.775919 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:57.776416 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:57.776421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:57.778180 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
W20260811 02:37:57.778419 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:57.773746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:60849 (local address 127.2.251.62:35649)
0811 02:37:57.773817 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:57.773819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:57.773829 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:37:57.773894 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:57.773895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:57.773907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:57.773955 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:57.774294 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.774802 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.775561 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:57.775750 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:57.775806 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:57.778327 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:60849: BlockingRecv error: recv got EOF from 127.2.251.3:60849 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 11 02:37:58 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:58 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:58.481164 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:58.474896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:36657)
0811 02:37:58.475071 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:58.475085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:58.475148 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:58.475373 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:58.475375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:58.475569 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:58.475620 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:58.475622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.476467 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.476468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:58.477158 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:58.477162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.477631 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.477632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:58.477702 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:58.478278 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:58.478283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:58.481057 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
W20260811 02:37:58.481354 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:58.475044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:36657 (local address 127.2.251.62:35649)
0811 02:37:58.475170 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:58.475172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:58.475183 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:37:58.475213 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:58.475214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:58.475225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:58.475326 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:58.475731 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.476336 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.477281 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.477546 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.477616 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:58.481257 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:36657: BlockingRecv error: recv got EOF from 127.2.251.1:36657 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
Aug 11 02:37:58 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:58 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:58.627586 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:58.621380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:56109)
0811 02:37:58.621680 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:58.621697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:58.621863 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:58.622070 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:58.622071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:58.622314 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:58.622388 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:58.622391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.623171 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.623172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:58.623773 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:58.623776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.624145 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.624146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:58.624205 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:58.624871 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:58.624878 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:58.627470 (+ 2592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
W20260811 02:37:58.627704 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:58.621742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:56109 (local address 127.2.251.62:35649)
0811 02:37:58.621918 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:58.621921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:58.621930 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:37:58.621978 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:58.621979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:58.621994 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:58.622050 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:58.622476 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.623074 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.623867 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.624074 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.624138 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:58.627641 (+ 3503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:56109: BlockingRecv error: recv got EOF from 127.2.251.2:56109 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 11 02:37:58 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:58 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:58.784535 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:58.778862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:57277)
0811 02:37:58.779031 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:58.779043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:58.779124 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:58.779303 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:58.779305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:58.779503 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:58.779557 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:58.779559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.780340 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.780342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:58.780915 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:58.780917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.781447 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.781448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:58.781502 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:58.782054 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:58.782058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:58.784417 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
W20260811 02:37:58.784667 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:58.778988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:57277 (local address 127.2.251.62:35649)
0811 02:37:58.779058 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:58.779060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:58.779105 (+ 45us) server_negotiation.cc:407] Connection header received
0811 02:37:58.779181 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:58.779182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:58.779195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:58.779245 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:58.779680 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.780232 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.781076 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:58.781323 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:58.781429 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:58.784585 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:57277: BlockingRecv error: recv got EOF from 127.2.251.3:57277 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 11 02:37:59 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:59 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:59.487293 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:59.482024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:39415)
0811 02:37:59.482275 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:59.482300 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:59.482401 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:59.482605 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:59.482606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:59.482790 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:59.482890 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:59.482891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.483623 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.483624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:59.484192 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:59.484194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.484586 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.484587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:59.484639 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:59.485157 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:59.485162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:59.487195 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
W20260811 02:37:59.487488 18384 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:59.482227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:39415 (local address 127.2.251.62:35649)
0811 02:37:59.482456 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:59.482457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:59.482472 (+ 15us) server_negotiation.cc:407] Connection header received
0811 02:37:59.482500 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:59.482500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:59.482512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:59.482591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:59.482987 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.483519 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.484275 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.484492 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.484551 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:59.487417 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:39415: BlockingRecv error: recv got EOF from 127.2.251.1:39415 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":118,"threads_started":1}
Aug 11 02:37:59 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:59 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:59.634112 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:59.628520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:55561)
0811 02:37:59.628752 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:59.628768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:59.628895 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:59.629067 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:59.629068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:59.629300 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:59.629367 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:59.629369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.630145 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.630147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:59.630840 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:59.630849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.631324 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.631325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:59.631381 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:59.631940 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:59.631946 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:59.634024 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
W20260811 02:37:59.634238 18384 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:59.628707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:55561 (local address 127.2.251.62:35649)
0811 02:37:59.628796 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:59.628798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:59.628874 (+ 76us) server_negotiation.cc:407] Connection header received
0811 02:37:59.628971 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:59.628973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:59.628990 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:59.629039 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:59.629452 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.630058 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.631032 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.631247 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.631313 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:59.634178 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:55561: BlockingRecv error: recv got EOF from 127.2.251.2:55561 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 02:37:59 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:37:59 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:37:59.791038 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:37:59.785093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:42055)
0811 02:37:59.785244 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 02:37:59.785254 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:37:59.785312 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:37:59.785444 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:37:59.785444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:37:59.785623 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:37:59.785675 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:59.785676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.786382 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.786383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:59.786930 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:37:59.786932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.787529 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.787532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:37:59.787612 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:59.788368 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:37:59.788376 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:37:59.790960 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
W20260811 02:37:59.791217 18384 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:37:59.785182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:42055 (local address 127.2.251.62:35649)
0811 02:37:59.785239 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:37:59.785241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:37:59.785299 (+ 58us) server_negotiation.cc:407] Connection header received
0811 02:37:59.785372 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:37:59.785373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:37:59.785385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:37:59.785436 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:37:59.785766 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.786287 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.787102 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:37:59.787349 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:37:59.787457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:37:59.791149 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:42055: BlockingRecv error: recv got EOF from 127.2.251.3:42055 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 11 02:38:00 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:00 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:00.493866 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:00.488032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:55727)
0811 02:38:00.488192 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:00.488203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:00.488297 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:00.488617 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:00.488619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:00.488917 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:00.489003 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:00.489006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.489958 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.489960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:00.490604 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:00.490608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.491060 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.491061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:00.491125 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:00.491872 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:00.491879 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:00.493800 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
W20260811 02:38:00.494230 18392 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:00.488152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:55727 (local address 127.2.251.62:35649)
0811 02:38:00.488333 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:00.488335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:00.488356 (+ 21us) server_negotiation.cc:407] Connection header received
0811 02:38:00.488396 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:00.488398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:00.488412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:00.488507 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:00.489150 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.489860 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.490709 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.490935 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.491006 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:00.494104 (+ 3098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:55727: BlockingRecv error: recv got EOF from 127.2.251.1:55727 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
Aug 11 02:38:00 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:00 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:00.640074 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:00.634858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:43589)
0811 02:38:00.635019 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:00.635032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:00.635101 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:00.635239 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:00.635240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:00.635443 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:00.635497 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:00.635499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.636168 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.636169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:00.636816 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:00.636821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.637287 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.637288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:00.637354 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:00.637929 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:00.637935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:00.639988 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260811 02:38:00.640257 18392 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:00.635034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:43589 (local address 127.2.251.62:35649)
0811 02:38:00.635110 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:00.635112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:00.635124 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:38:00.635160 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:00.635161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:00.635174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:00.635223 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:00.635579 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.636085 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.636942 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.637167 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.637241 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:00.640120 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:43589: BlockingRecv error: recv got EOF from 127.2.251.2:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 02:38:00 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:00 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:00.796757 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:00.791663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:56061)
0811 02:38:00.791851 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:00.791865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:00.791953 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:00.792099 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:00.792099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:00.792295 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:00.792352 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:00.792355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.793096 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.793097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:00.793639 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:00.793641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.794044 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.794045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:00.794095 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:00.794585 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:00.794590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:00.796688 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
W20260811 02:38:00.796854 18392 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:00.791759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:56061 (local address 127.2.251.62:35649)
0811 02:38:00.791856 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:00.791858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:00.791929 (+ 71us) server_negotiation.cc:407] Connection header received
0811 02:38:00.792018 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:00.792019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:00.792035 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:00.792082 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:00.792475 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.792982 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.793760 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:00.793965 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:00.794023 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:00.796789 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:56061: BlockingRecv error: recv got EOF from 127.2.251.3:56061 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 11 02:38:01 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:01 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:01.501889 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:01.494483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:33585)
0811 02:38:01.494631 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:01.494643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:01.494711 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:01.494886 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:01.494888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:01.495081 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:01.495135 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:01.495137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.495802 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.495803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:01.496325 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:01.496326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.496691 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.496692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:01.496734 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:01.499854 (+ 3120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:01.499859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:01.501823 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
W20260811 02:38:01.502020 18400 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:01.494540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:33585 (local address 127.2.251.62:35649)
0811 02:38:01.494688 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:01.494690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:01.494699 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:38:01.494780 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:01.494781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:01.494794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:01.494863 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:01.495199 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.495691 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.496434 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.496630 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.496683 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:01.501948 (+ 5265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:33585: BlockingRecv error: recv got EOF from 127.2.251.1:33585 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
Aug 11 02:38:01 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:01 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:01.647138 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:01.640772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:35451)
0811 02:38:01.640996 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:01.641072 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:01.641158 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:01.641457 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:01.641459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:01.641702 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:01.641837 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:01.641840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.642665 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.642667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:01.643280 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:01.643283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.643676 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.643678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:01.643751 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:01.644703 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:01.644708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:01.647072 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
W20260811 02:38:01.647343 18400 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:01.640893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:35451 (local address 127.2.251.62:35649)
0811 02:38:01.640947 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:01.640950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:01.641175 (+ 225us) server_negotiation.cc:407] Connection header received
0811 02:38:01.641325 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:01.641326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:01.641341 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:01.641412 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:01.641946 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.642539 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.643381 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.643584 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.643656 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:01.647182 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:35451: BlockingRecv error: recv got EOF from 127.2.251.2:35451 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 11 02:38:01 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:01 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:01.801956 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:01.797246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:58527)
0811 02:38:01.797440 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:01.797452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:01.797525 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:01.797639 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:01.797639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:01.797829 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:01.797877 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:01.797879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.798517 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.798518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:01.799063 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:01.799064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.799440 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.799441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:01.799508 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:01.799943 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:01.799948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:01.801871 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":103,"threads_started":1}
W20260811 02:38:01.802021 18400 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:01.797303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:58527 (local address 127.2.251.62:35649)
0811 02:38:01.797368 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:01.797370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:01.797537 (+ 167us) server_negotiation.cc:407] Connection header received
0811 02:38:01.797565 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:01.797566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:01.797579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:01.797628 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:01.797928 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.798426 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.799148 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:01.799345 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:01.799406 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:01.801963 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:58527: BlockingRecv error: recv got EOF from 127.2.251.3:58527 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 11 02:38:02 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:02 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:02.508718 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:02.502598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:54139)
0811 02:38:02.502902 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:02.502921 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:02.503062 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:02.503216 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:02.503217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:02.503423 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:02.503474 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:02.503476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.504432 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.504433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:02.505150 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:02.505157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.505650 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.505652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:02.505751 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:02.506456 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:02.506463 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:02.508465 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
W20260811 02:38:02.508867 18407 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:02.502917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:54139 (local address 127.2.251.62:35649)
0811 02:38:02.503092 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:02.503094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:02.503104 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:38:02.503135 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:02.503136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:02.503147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:02.503204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:02.503560 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.504365 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.505320 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.505572 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.505642 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:02.508790 (+ 3148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:54139: BlockingRecv error: recv got EOF from 127.2.251.1:54139 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
Aug 11 02:38:02 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:02 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:02.654389 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:02.647877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:60487)
0811 02:38:02.648041 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:02.648053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:02.648312 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:02.648906 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:02.648909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:02.649158 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:02.649241 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:02.649244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.650276 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.650278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:02.650872 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:02.650874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.651406 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.651408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:02.651496 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:02.652169 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:02.652174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:02.654277 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
W20260811 02:38:02.654487 18407 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:02.648011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:60487 (local address 127.2.251.62:35649)
0811 02:38:02.648072 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:02.648074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:02.648318 (+ 244us) server_negotiation.cc:407] Connection header received
0811 02:38:02.648610 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:02.648611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:02.648625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:02.648811 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:02.649409 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.650147 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.651012 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.651272 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.651360 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:02.654420 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:60487: BlockingRecv error: recv got EOF from 127.2.251.2:60487 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 11 02:38:02 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:02 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:02.809688 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:02.802634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:33435)
0811 02:38:02.802775 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:02.802788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:02.802957 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:02.803199 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:02.803201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:02.803534 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:02.803620 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:02.803623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.804690 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.804692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:02.805462 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:02.805469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.806001 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.806002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:02.806067 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:02.806671 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:02.806676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:02.809579 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
W20260811 02:38:02.809891 18407 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:02.802764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:33435 (local address 127.2.251.62:35649)
0811 02:38:02.803036 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:02.803039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:02.803069 (+ 30us) server_negotiation.cc:407] Connection header received
0811 02:38:02.803102 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:02.803103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:02.803125 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:02.803176 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:02.803733 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.804577 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.805607 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:02.805923 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:02.805996 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:02.809817 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:33435: BlockingRecv error: recv got EOF from 127.2.251.3:33435 (error 108)
Metrics: {"server-negotiator.queue_time_us":218}
Aug 11 02:38:03 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:03 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:03.515133 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:03.509392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:34853)
0811 02:38:03.509542 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:03.509551 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:03.509601 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:03.509973 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:03.509975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:03.510153 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:03.510214 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:03.510216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.510983 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.510984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:03.511523 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:03.511524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.511992 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.511994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:03.512076 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:03.512927 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:03.512933 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:03.515067 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
W20260811 02:38:03.515302 18411 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:03.509588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:34853 (local address 127.2.251.62:35649)
0811 02:38:03.509769 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:03.509771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:03.509779 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:38:03.509806 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:03.509807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:03.509819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:03.509907 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:03.510342 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.510872 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.511639 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.511853 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.511917 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:03.515226 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:34853: BlockingRecv error: recv got EOF from 127.2.251.1:34853 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
Aug 11 02:38:03 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:03 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:03.660068 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:03.655055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:39619)
0811 02:38:03.655234 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:03.655249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:03.655364 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:03.655512 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:03.655513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:03.655705 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:03.655762 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:03.655764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.656589 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.656590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:03.657126 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:03.657128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.657494 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.657495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:03.657553 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:03.658138 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:03.658144 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:03.659983 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
W20260811 02:38:03.660143 18411 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:03.655199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:39619 (local address 127.2.251.62:35649)
0811 02:38:03.655264 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:03.655267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:03.655339 (+ 72us) server_negotiation.cc:407] Connection header received
0811 02:38:03.655431 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:03.655432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:03.655445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:03.655498 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:03.655850 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.656498 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.657232 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.657423 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.657482 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:03.660088 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:39619: BlockingRecv error: recv got EOF from 127.2.251.2:39619 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 11 02:38:03 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:03 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:03.816366 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:03.810466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:46041)
0811 02:38:03.810793 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:03.810810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:03.810909 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:03.811062 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:03.811063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:03.811263 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:03.811335 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:03.811337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.812182 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.812185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:03.812819 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:03.812821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.813363 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.813365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:03.813458 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:03.814156 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:03.814161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:03.816277 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":192,"threads_started":1}
W20260811 02:38:03.816515 18411 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:03.810658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:46041 (local address 127.2.251.62:35649)
0811 02:38:03.810787 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:03.810790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:03.810894 (+ 104us) server_negotiation.cc:407] Connection header received
0811 02:38:03.810976 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:03.810978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:03.810994 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:03.811046 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:03.811422 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.811935 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.812926 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:03.813144 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:03.813339 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:03.816457 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:46041: BlockingRecv error: recv got EOF from 127.2.251.3:46041 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 11 02:38:04 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:04 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:04.522171 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:04.515881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:60361)
0811 02:38:04.516089 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:04.516101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:04.516166 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:04.516360 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:04.516361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:04.516538 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:04.516592 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:04.516594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.517417 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.517420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:04.518203 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:04.518208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.518646 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.518647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:04.518703 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:04.519577 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:04.519582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:04.522099 (+ 2517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":117,"threads_started":1}
W20260811 02:38:04.522343 18423 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:04.516033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:60361 (local address 127.2.251.62:35649)
0811 02:38:04.516159 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:04.516161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:04.516171 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:38:04.516215 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:04.516216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:04.516228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:04.516289 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:04.516702 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.517254 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.518317 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.518545 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.518612 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:04.522263 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:60361: BlockingRecv error: recv got EOF from 127.2.251.1:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
Aug 11 02:38:04 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:04 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:04.666579 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:04.660563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:41783)
0811 02:38:04.660698 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:04.660709 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:04.660785 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:04.660941 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:04.660942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:04.661137 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:04.661205 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:04.661207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.662092 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.662095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:04.662767 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:04.662771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.663387 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.663388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:04.663465 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:04.664128 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:04.664133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:04.666499 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260811 02:38:04.666733 18423 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:04.660628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:41783 (local address 127.2.251.62:35649)
0811 02:38:04.660691 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:04.660693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:04.660777 (+ 84us) server_negotiation.cc:407] Connection header received
0811 02:38:04.660845 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:04.660846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:04.660860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:04.660925 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:04.661274 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.661885 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.663028 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.663302 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.663372 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:04.666665 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:41783: BlockingRecv error: recv got EOF from 127.2.251.2:41783 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 11 02:38:04 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:04 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 02:38:04.823324 18423 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:04.817157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:51687 (local address 127.2.251.62:35649)
0811 02:38:04.817269 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:04.817272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:04.817403 (+ 131us) server_negotiation.cc:407] Connection header received
0811 02:38:04.817447 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:04.817448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:04.817464 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:04.817520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:04.818023 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.818574 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.819372 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.819588 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.819650 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:04.823265 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:51687: BlockingRecv error: recv got EOF from 127.2.251.3:51687 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 02:38:04.823311 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:04.817043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:51687)
0811 02:38:04.817291 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:04.817304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:04.817399 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:04.817601 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:04.817602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:04.817812 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:04.817865 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:04.817867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.818676 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.818678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:04.819272 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:04.819274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:04.819685 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:04.819686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:04.819763 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:04.820571 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:04.820579 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:04.823119 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":177,"threads_started":1}
Aug 11 02:38:05 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:05 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:05.527971 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:05.522818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:43759)
0811 02:38:05.522950 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:05.522963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:05.523020 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:05.523183 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:05.523183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:05.523358 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:05.523417 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:05.523418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.524103 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.524104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:05.524631 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:05.524632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.525001 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.525002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:05.525040 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:05.525549 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:05.525556 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:05.527906 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
W20260811 02:38:05.528190 18427 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:05.522910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:43759 (local address 127.2.251.62:35649)
0811 02:38:05.523051 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:05.523052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:05.523061 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:38:05.523085 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:05.523086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:05.523102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:05.523168 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:05.523502 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.524018 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.524715 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.524928 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.524992 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:05.528118 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:43759: BlockingRecv error: recv got EOF from 127.2.251.1:43759 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
Aug 11 02:38:05 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:05 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:05.672916 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:05.667194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:42329)
0811 02:38:05.667338 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:05.667357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:05.667414 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:05.667787 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:05.667788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:05.668201 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:05.668375 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:05.668380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.669119 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.669120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:05.669807 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:05.669811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.670235 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.670236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:05.670292 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:05.670799 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:05.670804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:05.672836 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
W20260811 02:38:05.673017 18427 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:05.667319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:42329 (local address 127.2.251.62:35649)
0811 02:38:05.667420 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:05.667421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:05.667433 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:38:05.667464 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:05.667465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:05.667475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:05.667754 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:05.668498 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.669027 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.669950 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.670140 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.670194 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:05.672961 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:42329: BlockingRecv error: recv got EOF from 127.2.251.2:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 11 02:38:05 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:05 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:05.830226 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:05.824010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:38353)
0811 02:38:05.824278 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:05.824312 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:05.824420 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:05.824778 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:05.824779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:05.825008 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:05.825081 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:05.825083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.825877 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.825878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:05.826440 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:05.826442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.827041 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.827043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:05.827101 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:05.827735 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:05.827741 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:05.830136 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
W20260811 02:38:05.830384 18427 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:05.824246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:38353 (local address 127.2.251.62:35649)
0811 02:38:05.824389 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:05.824391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:05.824406 (+ 15us) server_negotiation.cc:407] Connection header received
0811 02:38:05.824504 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:05.824505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:05.824520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:05.824575 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:05.825179 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.825691 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.826577 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:05.826900 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:05.826989 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:05.830255 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:38353: BlockingRecv error: recv got EOF from 127.2.251.3:38353 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 11 02:38:06 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:06 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:06.537277 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:06.528693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:60677)
0811 02:38:06.528920 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:06.528934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:06.529154 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:06.529411 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:06.529413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:06.529603 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:06.529651 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:06.529652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.530401 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.530402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:06.530948 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:06.530950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.531315 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.531316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:06.531354 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:06.535206 (+ 3852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:06.535214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:06.537177 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
W20260811 02:38:06.537362 18437 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:06.528854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:60677 (local address 127.2.251.62:35649)
0811 02:38:06.529040 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:06.529042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:06.529109 (+ 67us) server_negotiation.cc:407] Connection header received
0811 02:38:06.529231 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:06.529240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:06.529253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:06.529344 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:06.529743 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.530293 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.531049 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.531251 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.531312 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:06.537295 (+ 5983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:60677: BlockingRecv error: recv got EOF from 127.2.251.1:60677 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
Aug 11 02:38:06 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:06 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:06.678989 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:06.673511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:40767)
0811 02:38:06.673705 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:06.673741 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:06.673812 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:06.673940 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:06.673940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:06.674115 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:06.674164 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:06.674166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.675162 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.675164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:06.675782 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:06.675784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.676258 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.676260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:06.676345 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:06.676870 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:06.676875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:06.678906 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
W20260811 02:38:06.679090 18437 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:06.673635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:40767 (local address 127.2.251.62:35649)
0811 02:38:06.673704 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:06.673706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:06.673797 (+ 91us) server_negotiation.cc:407] Connection header received
0811 02:38:06.673873 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:06.673873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:06.673885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:06.673924 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:06.674327 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.675016 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.675917 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.676131 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.676201 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:06.679026 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:40767: BlockingRecv error: recv got EOF from 127.2.251.2:40767 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 11 02:38:06 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:06 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:06.836143 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:06.830956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:32885)
0811 02:38:06.831112 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:06.831125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:06.831189 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:06.831396 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:06.831397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:06.831615 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:06.831670 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:06.831671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.832447 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.832448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:06.833127 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:06.833130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.833513 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.833514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:06.833561 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:06.834048 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:06.834053 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:06.836031 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
W20260811 02:38:06.836338 18437 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:06.831100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:32885 (local address 127.2.251.62:35649)
0811 02:38:06.831194 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:06.831196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:06.831207 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:38:06.831241 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:06.831242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:06.831256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:06.831327 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:06.831821 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.832352 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.833227 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:06.833412 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:06.833487 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:06.836188 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:32885: BlockingRecv error: recv got EOF from 127.2.251.3:32885 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 11 02:38:07 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:07 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:07.543744 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:07.538031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:55217)
0811 02:38:07.538179 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:07.538191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:07.538251 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:07.538539 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:07.538541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:07.538745 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:07.538800 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:07.538802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.539550 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.539552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:07.540119 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:07.540121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.540595 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.540597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:07.540649 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:07.541177 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:07.541182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:07.543640 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
W20260811 02:38:07.543829 18454 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:07.538153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:55217 (local address 127.2.251.62:35649)
0811 02:38:07.538330 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:07.538332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:07.538352 (+ 20us) server_negotiation.cc:407] Connection header received
0811 02:38:07.538383 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:07.538384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:07.538396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:07.538484 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:07.538883 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.539413 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.540240 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.540456 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.540522 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:07.543760 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:55217: BlockingRecv error: recv got EOF from 127.2.251.1:55217 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
Aug 11 02:38:07 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:07 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:07.685422 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:07.679546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:52075)
0811 02:38:07.679684 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:07.679696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:07.679775 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:07.679958 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:07.679959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:07.680138 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:07.680187 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:07.680189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.681116 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.681117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:07.681839 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:07.681843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.682337 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.682338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:07.682390 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:07.682888 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:07.682893 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:07.685286 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260811 02:38:07.685650 18454 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:07.679615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52075 (local address 127.2.251.62:35649)
0811 02:38:07.679668 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:07.679670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:07.679830 (+ 160us) server_negotiation.cc:407] Connection header received
0811 02:38:07.679868 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:07.679869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:07.679884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:07.679945 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:07.680278 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.680969 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.681951 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.682229 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.682312 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:07.685550 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:52075: BlockingRecv error: recv got EOF from 127.2.251.2:52075 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 11 02:38:07 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:07 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:07.842355 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:07.836811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:59959)
0811 02:38:07.836976 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:07.836989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:07.837045 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:07.837215 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:07.837216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:07.837400 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:07.837450 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:07.837452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.838351 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.838355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:07.839345 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:07.839350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.839710 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.839711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:07.839766 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:07.840248 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:07.840253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:07.842218 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
W20260811 02:38:07.842389 18454 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:07.836971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:59959 (local address 127.2.251.62:35649)
0811 02:38:07.837071 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:07.837073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:07.837086 (+ 13us) server_negotiation.cc:407] Connection header received
0811 02:38:07.837119 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:07.837120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:07.837133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:07.837201 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:07.837533 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.838204 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.839429 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:07.839614 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:07.839673 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:07.842330 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:59959: BlockingRecv error: recv got EOF from 127.2.251.3:59959 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 11 02:38:08 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:08 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:08.549652 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:08.544462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:33541)
0811 02:38:08.544679 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:08.544694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:08.544881 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:08.545111 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:08.545112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:08.545317 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:08.545366 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:08.545368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.546118 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.546119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:08.546692 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:08.546694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.547089 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.547090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:08.547206 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:08.547727 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:08.547733 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:08.549554 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":146,"threads_started":1}
W20260811 02:38:08.549782 18461 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:08.544497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:33541 (local address 127.2.251.62:35649)
0811 02:38:08.544693 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:08.544695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:08.544883 (+ 188us) server_negotiation.cc:407] Connection header received
0811 02:38:08.544952 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:08.544953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:08.544969 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:08.545048 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:08.545464 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.546020 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.546780 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.546983 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.547047 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:08.549667 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:33541: BlockingRecv error: recv got EOF from 127.2.251.1:33541 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
Aug 11 02:38:08 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:08 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:08.691699 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:08.686236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:34299)
0811 02:38:08.686391 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:08.686402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:08.686460 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:08.686610 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:08.686612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:08.686805 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:08.686858 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:08.686860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.687803 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.687805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:08.688385 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:08.688387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.688808 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.688810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:08.688874 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:08.689429 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:08.689438 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:08.691613 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
W20260811 02:38:08.691833 18461 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:08.686427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:34299 (local address 127.2.251.62:35649)
0811 02:38:08.686496 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:08.686499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:08.686507 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:38:08.686539 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:08.686540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:08.686551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:08.686599 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:08.686934 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.687701 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.688468 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.688674 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.688769 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:08.691760 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:34299: BlockingRecv error: recv got EOF from 127.2.251.2:34299 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 02:38:08 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:08 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:08.849453 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:08.842876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:34593)
0811 02:38:08.843035 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:08.843048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:08.843131 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:08.843378 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:08.843379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:08.843563 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:08.843619 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:08.843621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.844406 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.844407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:08.844963 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:08.844964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.845470 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.845471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:08.845559 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:08.846186 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:08.846191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:08.849343 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
W20260811 02:38:08.849859 18461 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:08.842956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:34593 (local address 127.2.251.62:35649)
0811 02:38:08.843022 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:08.843024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:08.843213 (+ 189us) server_negotiation.cc:407] Connection header received
0811 02:38:08.843260 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:08.843262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:08.843287 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:08.843357 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:08.843744 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.844298 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.845097 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:08.845386 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:08.845468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:08.849537 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:34593: BlockingRecv error: recv got EOF from 127.2.251.3:34593 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 02:38:09 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:09 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:09.555824 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:09.550401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:50477)
0811 02:38:09.550544 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:09.550555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:09.550612 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:09.550849 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:09.550851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:09.551122 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:09.551184 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:09.551187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.551969 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.551970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:09.552554 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:09.552556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.553000 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.553001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:09.553070 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:09.553564 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:09.553569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:09.555757 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
W20260811 02:38:09.555979 18469 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:09.550506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:50477 (local address 127.2.251.62:35649)
0811 02:38:09.550665 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:09.550667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:09.550678 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:38:09.550707 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:09.550708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:09.550718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:09.550799 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:09.551316 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.551870 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.552640 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.552906 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.552980 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:09.555896 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:50477: BlockingRecv error: recv got EOF from 127.2.251.1:50477 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
Aug 11 02:38:09 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:09 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:09.697268 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:09.692335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:59973)
0811 02:38:09.692517 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:09.692528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:09.692596 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:09.692880 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:09.692881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:09.693061 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:09.693116 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:09.693117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.693835 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.693836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:09.694365 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:09.694367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.694775 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.694776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:09.694835 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:09.695331 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:09.695336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:09.697200 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
W20260811 02:38:09.697382 18469 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:09.692396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:59973 (local address 127.2.251.62:35649)
0811 02:38:09.692453 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:09.692455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:09.692640 (+ 185us) server_negotiation.cc:407] Connection header received
0811 02:38:09.692754 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:09.692755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:09.692768 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:09.692816 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:09.693218 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.693759 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.694470 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.694680 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.694744 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:09.697317 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:59973: BlockingRecv error: recv got EOF from 127.2.251.2:59973 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 11 02:38:09 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:09 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.3@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:09.855131 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:09.850123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.3:58403)
0811 02:38:09.850306 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:09.850320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:09.850399 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:09.850582 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:09.850583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:09.850759 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:09.850836 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:09.850838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.851506 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.851507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:09.852053 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:09.852055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.852486 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.852487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:09.852542 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:09.853087 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:09.853092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:09.855050 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
W20260811 02:38:09.855257 18469 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:09.850191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:58403 (local address 127.2.251.62:35649)
0811 02:38:09.850265 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:09.850267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:09.850443 (+ 176us) server_negotiation.cc:407] Connection header received
0811 02:38:09.850473 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:09.850474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:09.850488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:09.850529 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:09.850897 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.851417 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.852152 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:09.852357 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:09.852417 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:09.855181 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.3:58403: BlockingRecv error: recv got EOF from 127.2.251.3:58403 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 02:38:10 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:10 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.1@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:10.561509 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:10.556528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.1:54721)
0811 02:38:10.556698 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:10.556716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:10.556793 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:10.556971 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:10.556972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:10.557148 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:10.557201 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:10.557202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.557952 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.557953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:10.558507 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:10.558509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.558925 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.558926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:10.558977 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:10.559557 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:10.559561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:10.561429 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
W20260811 02:38:10.561992 18476 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:10.556680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:54721 (local address 127.2.251.62:35649)
0811 02:38:10.556801 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:10.556804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:10.556816 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:38:10.556845 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:10.556846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:10.556858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:10.556921 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:10.557265 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.557850 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.558616 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.558839 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.558903 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:10.561893 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.1:54721: BlockingRecv error: recv got EOF from 127.2.251.1:54721 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
Aug 11 02:38:10 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 02:38:10 dist-test-slave-cvwc krb5kdc[16579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.251.2@KRBTEST.COM for oryx/127.2.251.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 02:38:10.704481 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:10.698043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35649 (local address 127.2.251.2:50203)
0811 02:38:10.698198 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:10.698210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:10.698270 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:10.698498 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:10.698499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:10.698679 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:10.698733 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:10.698735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.699524 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.699525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:10.700063 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:10.700064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.700555 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.700556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:10.700612 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:10.701225 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:10.701233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:10.704390 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.251.62:35649: Server oryx/127.2.251.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
W20260811 02:38:10.704795 18476 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:10.698181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:50203 (local address 127.2.251.62:35649)
0811 02:38:10.698283 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:10.698285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:10.698296 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:38:10.698347 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:10.698348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:10.698360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:10.698431 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:10.698868 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.699389 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.700217 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:10.700441 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:10.700509 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:10.704609 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.251.2:50203: BlockingRecv error: recv got EOF from 127.2.251.2:50203 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 02:38:10.747587 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 17596
I20260811 02:38:10.782958 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 17875
I20260811 02:38:10.811587 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 18132
I20260811 02:38:10.823020 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 17469
2026-08-11T02:38:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18022 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T02:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:38:10Z Disabled control of system clock
I20260811 02:38:10.875887 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38147
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:38147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 02:38:10.967960 18483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:10.968235 18483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:10.968261 18483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:10.969507 18483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:38:10.969560 18483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:10.969574 18483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:10.969588 18483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:38:10.969601 18483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:38:10.971251 18483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:38147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:10.971510 18483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:10.971777 18483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:10.977512 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:10.977546 18492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:10.977535 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:10.977553 18483 server_base.cc:1034] running on GCE node
I20260811 02:38:10.978178 18483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:10.978507 18483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:10.979682 18483 hybrid_clock.cc:648] HybridClock initialized: now 1786415890979659 us; error 40 us; skew 500 ppm
I20260811 02:38:10.981091 18483 webserver.cc:460] Webserver started at http://127.2.251.62:33971/ using document root <none> and password file <none>
I20260811 02:38:10.981352 18483 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:10.981406 18483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:10.981549 18483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:10.982946 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "82f266b834e7405e90b5229eeba3d145"
format_stamp: "Formatted at 2026-08-11 02:38:10 on dist-test-slave-cvwc"
server_key: "bbe4f0a7a566bf09ba8a1cb003200db7"
server_key_iv: "9a52dad9a9c7335d660fdbcaa186c24e"
server_key_version: "encryptionkey@0"
I20260811 02:38:10.983350 18483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "82f266b834e7405e90b5229eeba3d145"
format_stamp: "Formatted at 2026-08-11 02:38:10 on dist-test-slave-cvwc"
server_key: "bbe4f0a7a566bf09ba8a1cb003200db7"
server_key_iv: "9a52dad9a9c7335d660fdbcaa186c24e"
server_key_version: "encryptionkey@0"
I20260811 02:38:10.984920 18483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:38:10.986289 18509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:10.986507 18483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:38:10.986609 18483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "82f266b834e7405e90b5229eeba3d145"
format_stamp: "Formatted at 2026-08-11 02:38:10 on dist-test-slave-cvwc"
server_key: "bbe4f0a7a566bf09ba8a1cb003200db7"
server_key_iv: "9a52dad9a9c7335d660fdbcaa186c24e"
server_key_version: "encryptionkey@0"
I20260811 02:38:10.986685 18483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:10.999831 18483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:11.000223 18483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:11.000372 18483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:11.004658 18483 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:38147
I20260811 02:38:11.004695 18605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:38147 every 8 connection(s)
I20260811 02:38:11.005052 18483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:38:11.006122 18606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:38:11.008946 18606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145: Bootstrap starting.
I20260811 02:38:11.009575 18606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145: Neither blocks nor log segments found. Creating new log.
I20260811 02:38:11.009895 18606 log.cc:824] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145: Log is configured to *not* fsync() on all Append() calls
I20260811 02:38:11.010658 18606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145: No bootstrap required, opened a new log
I20260811 02:38:11.011973 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 18483
I20260811 02:38:11.012074 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:38:11.012253 3052 external_mini_cluster.cc:1428] Setting key 91ceda8d8f4c952390a0369a290a279d
I20260811 02:38:11.012558 18606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f266b834e7405e90b5229eeba3d145" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38147 } }
I20260811 02:38:11.012814 18606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:38:11.012888 18606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f266b834e7405e90b5229eeba3d145, State: Initialized, Role: FOLLOWER
I20260811 02:38:11.013047 18606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f266b834e7405e90b5229eeba3d145" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38147 } }
I20260811 02:38:11.013128 18606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:38:11.013151 18606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:38:11.013206 18606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:38:11.013942 18606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f266b834e7405e90b5229eeba3d145" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38147 } }
I20260811 02:38:11.014036 18606 leader_election.cc:302] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82f266b834e7405e90b5229eeba3d145; no voters:
I20260811 02:38:11.014197 18606 leader_election.cc:288] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:38:11.014261 18613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:38:11.014469 18606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:38:11.014482 18613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [term 1 LEADER]: Becoming Leader. State: Replica: 82f266b834e7405e90b5229eeba3d145, State: Running, Role: LEADER
I20260811 02:38:11.014662 18613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f266b834e7405e90b5229eeba3d145" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38147 } }
I20260811 02:38:11.015069 18614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82f266b834e7405e90b5229eeba3d145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f266b834e7405e90b5229eeba3d145" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38147 } } }
I20260811 02:38:11.015144 18614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [sys.catalog]: This master's current role is: LEADER
I20260811 02:38:11.015336 18616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f266b834e7405e90b5229eeba3d145. Latest consensus state: current_term: 1 leader_uuid: "82f266b834e7405e90b5229eeba3d145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f266b834e7405e90b5229eeba3d145" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38147 } } }
I20260811 02:38:11.015430 18616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145 [sys.catalog]: This master's current role is: LEADER
W20260811 02:38:11.016382 18642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:38:11.016443 18642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:38:11.016742 18623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:38:11.016927 18623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:38:11.018808 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.012875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.2.251.62:38147)
0811 02:38:11.013124 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.013134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.013170 (+ 36us) server_negotiation.cc:407] Connection header received
0811 02:38:11.013302 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.013310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.013367 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.013500 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.013788 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.015572 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.016552 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.017138 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.017263 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.018093 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:11.018098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:11.018098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:11.018307 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:11.018350 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:11.018377 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:11.018442 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:11.018543 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 02:38:11.018628 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260811 02:38:11.019218 18623 catalog_manager.cc:1347] Generated new cluster ID: 573e19c28fb94a689fc204c0cfec393d
I20260811 02:38:11.019261 18623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:38:11.039024 18623 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:38:11.039563 18623 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:38:11.050858 18623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82f266b834e7405e90b5229eeba3d145: Generated new TSK 0
I20260811 02:38:11.051113 18623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:38:11.061034 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38147
--builtin_ntp_servers=127.2.251.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 02:38:11.154323 18645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:11.154559 18645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:11.154582 18645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:11.155812 18645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:11.155857 18645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:11.155882 18645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:38:11.157447 18645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:11.157706 18645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:11.158056 18645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:11.163424 18657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.163424 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.163447 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:11.163450 18645 server_base.cc:1034] running on GCE node
I20260811 02:38:11.163842 18645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:11.164078 18645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:11.165256 18645 hybrid_clock.cc:648] HybridClock initialized: now 1786415891165244 us; error 30 us; skew 500 ppm
I20260811 02:38:11.167145 18645 webserver.cc:460] Webserver started at http://127.2.251.1:42621/ using document root <none> and password file <none>
I20260811 02:38:11.167373 18645 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:11.167425 18645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:11.167716 18645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:11.168874 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "875f79a6228f4816a818356f68d67df0"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "9991d89fa6fa4d495b6605df7be45e79"
server_key_iv: "69bf72ac215c898329100780fd51c34e"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.169215 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "875f79a6228f4816a818356f68d67df0"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "9991d89fa6fa4d495b6605df7be45e79"
server_key_iv: "69bf72ac215c898329100780fd51c34e"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.170745 18645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:38:11.171588 18668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.171746 18645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:38:11.171831 18645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "875f79a6228f4816a818356f68d67df0"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "9991d89fa6fa4d495b6605df7be45e79"
server_key_iv: "69bf72ac215c898329100780fd51c34e"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.171897 18645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:11.192291 18645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:11.192633 18645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:11.192776 18645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:11.193102 18645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:11.193560 18645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:11.193598 18645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.193627 18645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:11.193645 18645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.199847 18645 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:45075
I20260811 02:38:11.199934 18866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:45075 every 8 connection(s)
I20260811 02:38:11.200241 18645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:38:11.206423 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.201115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:42889 (local address 127.2.251.62:38147)
0811 02:38:11.201275 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.201278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.201687 (+ 409us) server_negotiation.cc:407] Connection header received
0811 02:38:11.202301 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.202303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.202319 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.202465 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.203215 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.203747 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.204718 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.204909 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.204966 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.206073 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:11.206075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:11.206076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:11.206083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:11.206103 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:11.206109 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:11.206153 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:11.206245 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 02:38:11.206309 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 02:38:11.206790 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.201052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38147 (local address 127.2.251.1:42889)
0811 02:38:11.201407 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:11.201465 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:11.202175 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:11.202548 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:11.202552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:11.202562 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:11.203126 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:11.203132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.203840 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.203841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:11.204633 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:11.204636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.204986 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.204988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:11.205118 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.205814 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:38:11.205815 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:11.205871 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:38:11.205914 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:11.206163 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:11.206170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:11.206174 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:38:11.206247 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 02:38:11.206411 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260811 02:38:11.207379 18867 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38147
I20260811 02:38:11.207492 18867 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:11.207672 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 18645
I20260811 02:38:11.207728 18867 heartbeater.cc:511] Master 127.2.251.62:38147 requested a full tablet report, sending...
I20260811 02:38:11.207767 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:38:11.207971 3052 external_mini_cluster.cc:1428] Setting key b3bbf2b58cd06763714c2ff551ce7453
I20260811 02:38:11.208473 18543 ts_manager.cc:194] Registered new tserver with Master: 875f79a6228f4816a818356f68d67df0 (127.2.251.1:45075)
I20260811 02:38:11.209079 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38147
--builtin_ntp_servers=127.2.251.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 02:38:11.209095 18543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.1:42889
W20260811 02:38:11.303967 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:11.304222 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:11.304246 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:11.306028 18882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:11.306085 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:11.306113 18882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:38:11.307513 18882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:11.307729 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:11.308012 18882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:11.313578 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.313585 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.313664 18889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:11.313843 18882 server_base.cc:1034] running on GCE node
I20260811 02:38:11.314117 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:11.314363 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:11.315495 18882 hybrid_clock.cc:648] HybridClock initialized: now 1786415891315480 us; error 33 us; skew 500 ppm
I20260811 02:38:11.316906 18882 webserver.cc:460] Webserver started at http://127.2.251.2:36165/ using document root <none> and password file <none>
I20260811 02:38:11.317149 18882 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:11.317200 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:11.317319 18882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:11.318509 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "a405708658fb4546b97ea37345370a90"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "ca5dd523dde25f640234569cd11265a2"
server_key_iv: "fbd187549cbb684a2d90bf81d1d7284b"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.318854 18882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "a405708658fb4546b97ea37345370a90"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "ca5dd523dde25f640234569cd11265a2"
server_key_iv: "fbd187549cbb684a2d90bf81d1d7284b"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.320209 18882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:38:11.321017 18903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.321167 18882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260811 02:38:11.321254 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "a405708658fb4546b97ea37345370a90"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "ca5dd523dde25f640234569cd11265a2"
server_key_iv: "fbd187549cbb684a2d90bf81d1d7284b"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.321337 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:11.356009 18882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:11.356448 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:11.356616 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:11.356886 18882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:11.357259 18882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:11.357297 18882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.357328 18882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:11.357347 18882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.364477 18882 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:45499
I20260811 02:38:11.364526 19088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:45499 every 8 connection(s)
I20260811 02:38:11.364898 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:38:11.366283 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 18882
I20260811 02:38:11.366425 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:38:11.366636 3052 external_mini_cluster.cc:1428] Setting key e077ff09f7c8754e281e7cb6fb384f88
I20260811 02:38:11.368077 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38147
--builtin_ntp_servers=127.2.251.20:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 02:38:11.372051 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.365808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:59725 (local address 127.2.251.62:38147)
0811 02:38:11.365870 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.365873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.366373 (+ 500us) server_negotiation.cc:407] Connection header received
0811 02:38:11.367236 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.367237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.367252 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.367405 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.368569 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.369200 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.370558 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.370845 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.370932 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.371766 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:11.371768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:11.371768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:11.371774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:11.371798 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:11.371802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:11.371847 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:11.371933 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 02:38:11.371966 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 02:38:11.372279 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.365636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38147 (local address 127.2.251.2:59725)
0811 02:38:11.366055 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:11.366162 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:11.367035 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:11.367562 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:11.367568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:11.367580 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:11.368431 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:11.368439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.369363 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.369364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:11.370466 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:11.370470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.370945 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.370946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:11.371066 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.371620 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:38:11.371621 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:11.371651 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:38:11.371677 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:11.371854 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:11.371860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:11.371863 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:38:11.371931 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 02:38:11.372046 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":143,"threads_started":1}
I20260811 02:38:11.372787 19094 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38147
I20260811 02:38:11.372907 19094 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:11.373133 19094 heartbeater.cc:511] Master 127.2.251.62:38147 requested a full tablet report, sending...
I20260811 02:38:11.373592 18543 ts_manager.cc:194] Registered new tserver with Master: a405708658fb4546b97ea37345370a90 (127.2.251.2:45499)
I20260811 02:38:11.374135 18543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.2:59725
W20260811 02:38:11.463635 19099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:11.463876 19099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:11.463899 19099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:11.465117 19099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:11.465165 19099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:11.465190 19099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:38:11.466629 19099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:11.466836 19099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:11.467106 19099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:11.472884 19111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.472886 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.473006 19109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:11.473053 19099 server_base.cc:1034] running on GCE node
I20260811 02:38:11.473397 19099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:11.473685 19099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:11.474896 19099 hybrid_clock.cc:648] HybridClock initialized: now 1786415891474882 us; error 31 us; skew 500 ppm
I20260811 02:38:11.476464 19099 webserver.cc:460] Webserver started at http://127.2.251.3:34595/ using document root <none> and password file <none>
I20260811 02:38:11.476713 19099 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:11.476768 19099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:11.476923 19099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:11.478102 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "fe4ce4a299584a11a97f8821b14e1094"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "b669f1a3cb20c57f955b65e923190674"
server_key_iv: "5bb7dbcec118e48f21c4f612ba58add2"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.478472 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "fe4ce4a299584a11a97f8821b14e1094"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "b669f1a3cb20c57f955b65e923190674"
server_key_iv: "5bb7dbcec118e48f21c4f612ba58add2"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.479974 19099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:38:11.480842 19118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.481130 19099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:11.481253 19099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "fe4ce4a299584a11a97f8821b14e1094"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "b669f1a3cb20c57f955b65e923190674"
server_key_iv: "5bb7dbcec118e48f21c4f612ba58add2"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.481328 19099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:11.492796 19099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:11.493171 19099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:11.493326 19099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:11.493584 19099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:11.494007 19099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:11.494050 19099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.494084 19099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:11.494107 19099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.500490 19099 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:37069
I20260811 02:38:11.500592 19316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:37069 every 8 connection(s)
I20260811 02:38:11.500918 19099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:38:11.504890 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 19099
I20260811 02:38:11.504987 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:38:11.505148 3052 external_mini_cluster.cc:1428] Setting key 9c43db89e10aef55bf714fc309332c5e
I20260811 02:38:11.507669 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.502060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:49219 (local address 127.2.251.62:38147)
0811 02:38:11.502106 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.502108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.502507 (+ 399us) server_negotiation.cc:407] Connection header received
0811 02:38:11.503104 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.503107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.503126 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.503223 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.504070 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.504608 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.505929 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.506171 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.506238 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.507186 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:11.507187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:11.507187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:11.507193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:11.507207 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:11.507211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:11.507269 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:11.507560 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 02:38:11.507597 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 02:38:11.508037 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.502038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38147 (local address 127.2.251.3:49219)
0811 02:38:11.502346 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:11.502397 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:11.502916 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:11.503338 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:11.503342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:11.503353 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:11.503921 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:11.503926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.504748 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.504750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:11.505860 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:11.505864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:11.506257 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:11.506258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:11.506442 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:11.507018 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:38:11.507018 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:11.507051 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:38:11.507078 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:11.507361 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:11.507379 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:11.507385 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:38:11.507516 (+ 131us) client_negotiation.cc:240] Negotiation successful
0811 02:38:11.507696 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260811 02:38:11.508813 19317 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38147
I20260811 02:38:11.508960 19317 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:11.509186 19317 heartbeater.cc:511] Master 127.2.251.62:38147 requested a full tablet report, sending...
I20260811 02:38:11.509758 18543 ts_manager.cc:194] Registered new tserver with Master: fe4ce4a299584a11a97f8821b14e1094 (127.2.251.3:37069)
I20260811 02:38:11.510274 18543 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.3:49219
I20260811 02:38:11.517573 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 02:38:11.519523 18612 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:11.519023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.2.251.62:38147)
0811 02:38:11.519075 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.519077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.519147 (+ 70us) server_negotiation.cc:407] Connection header received
0811 02:38:11.519242 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.519243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.519257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.519308 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.519458 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53764: BlockingRecv error: recv got EOF from 127.0.0.1:53764 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:38:11.519598 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 18645
I20260811 02:38:11.540683 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 18882
I20260811 02:38:11.562381 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 19099
I20260811 02:38:11.576663 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 18483
2026-08-11T02:38:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (739 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T02:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:38:11Z Disabled control of system clock
I20260811 02:38:11.615224 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39171
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 02:38:11.717334 19334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:11.717587 19334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:11.717613 19334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:11.719007 19334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:38:11.719059 19334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:11.719071 19334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:11.719082 19334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:38:11.719094 19334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:38:11.720593 19334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:11.720885 19334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:11.721210 19334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:11.727453 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.727432 19352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.727423 19348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:11.727645 19334 server_base.cc:1034] running on GCE node
I20260811 02:38:11.728171 19334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:11.728515 19334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:11.729737 19334 hybrid_clock.cc:648] HybridClock initialized: now 1786415891729673 us; error 70 us; skew 500 ppm
I20260811 02:38:11.734670 19334 webserver.cc:460] Webserver started at http://127.2.251.62:42169/ using document root <none> and password file <none>
I20260811 02:38:11.734998 19334 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:11.735059 19334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:11.735209 19334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:11.736402 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "efa86ec797e44824956c415f66dc2e15"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "3083d25687d27d265b84ef96a5283b56"
server_key_iv: "5b87190ae4ec431fb49634d56749d592"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.736752 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "efa86ec797e44824956c415f66dc2e15"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "3083d25687d27d265b84ef96a5283b56"
server_key_iv: "5b87190ae4ec431fb49634d56749d592"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.738279 19334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:38:11.739379 19364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.739709 19334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:11.739835 19334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "efa86ec797e44824956c415f66dc2e15"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "3083d25687d27d265b84ef96a5283b56"
server_key_iv: "5b87190ae4ec431fb49634d56749d592"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.739938 19334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:11.743847 19334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:11.744158 19334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:11.744295 19334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:11.748603 19334 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39171
I20260811 02:38:11.748799 19457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39171 every 8 connection(s)
I20260811 02:38:11.749154 19334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:38:11.750011 19459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:38:11.751753 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 19334
I20260811 02:38:11.751874 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:38:11.752106 3052 external_mini_cluster.cc:1428] Setting key 1aa9f87cadf8570c71aec5bc8f02117c
I20260811 02:38:11.754093 19459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15: Bootstrap starting.
I20260811 02:38:11.754908 19459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15: Neither blocks nor log segments found. Creating new log.
I20260811 02:38:11.755203 19459 log.cc:824] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15: Log is configured to *not* fsync() on all Append() calls
I20260811 02:38:11.755621 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.753122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.2.251.62:39171)
0811 02:38:11.753468 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.753480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.753526 (+ 46us) server_negotiation.cc:407] Connection header received
0811 02:38:11.754128 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.754138 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.754199 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.754395 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.755123 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:11.755127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:11.755127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:11.755131 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:11.755174 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:11.755184 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:11.755260 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:11.755318 (+ 58us) server_negotiation.cc:299] Negotiation successful
0811 02:38:11.755393 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":142,"threads_started":1}
I20260811 02:38:11.756057 19459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15: No bootstrap required, opened a new log
I20260811 02:38:11.758090 19459 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa86ec797e44824956c415f66dc2e15" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39171 } }
I20260811 02:38:11.758246 19459 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:38:11.758270 19459 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa86ec797e44824956c415f66dc2e15, State: Initialized, Role: FOLLOWER
I20260811 02:38:11.758369 19459 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa86ec797e44824956c415f66dc2e15" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39171 } }
I20260811 02:38:11.758476 19459 raft_consensus.cc:388] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:38:11.758507 19459 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:38:11.758543 19459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:38:11.759111 19459 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa86ec797e44824956c415f66dc2e15" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39171 } }
I20260811 02:38:11.759204 19459 leader_election.cc:302] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efa86ec797e44824956c415f66dc2e15; no voters:
I20260811 02:38:11.759342 19459 leader_election.cc:288] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:38:11.759563 19464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:38:11.759666 19459 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:38:11.759727 19464 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [term 1 LEADER]: Becoming Leader. State: Replica: efa86ec797e44824956c415f66dc2e15, State: Running, Role: LEADER
I20260811 02:38:11.759845 19464 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa86ec797e44824956c415f66dc2e15" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39171 } }
I20260811 02:38:11.760226 19466 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efa86ec797e44824956c415f66dc2e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa86ec797e44824956c415f66dc2e15" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39171 } } }
I20260811 02:38:11.760334 19466 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [sys.catalog]: This master's current role is: LEADER
I20260811 02:38:11.760211 19467 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa86ec797e44824956c415f66dc2e15. Latest consensus state: current_term: 1 leader_uuid: "efa86ec797e44824956c415f66dc2e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa86ec797e44824956c415f66dc2e15" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39171 } } }
I20260811 02:38:11.760457 19467 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15 [sys.catalog]: This master's current role is: LEADER
I20260811 02:38:11.760563 19484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:38:11.761078 19484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:38:11.763164 19484 catalog_manager.cc:1347] Generated new cluster ID: cb538983eeca4ad7b5aabf8834d2997e
I20260811 02:38:11.763242 19484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:38:11.783903 19484 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:38:11.784125 19484 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:38:11.789825 19484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efa86ec797e44824956c415f66dc2e15: Generated new TSK 0
I20260811 02:38:11.790283 19484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:38:11.797078 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39171
--builtin_ntp_servers=127.2.251.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 02:38:11.897192 19501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:11.897437 19501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:11.897471 19501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:11.898783 19501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:11.898835 19501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:11.898880 19501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:38:11.900334 19501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:11.900609 19501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:11.900926 19501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:11.906502 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.906585 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:11.906891 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:11.906953 19501 server_base.cc:1034] running on GCE node
I20260811 02:38:11.907182 19501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:11.907454 19501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:11.908720 19501 hybrid_clock.cc:648] HybridClock initialized: now 1786415891908688 us; error 32 us; skew 500 ppm
I20260811 02:38:11.910808 19501 webserver.cc:460] Webserver started at http://127.2.251.1:43117/ using document root <none> and password file <none>
I20260811 02:38:11.911116 19501 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:11.911176 19501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:11.911324 19501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:11.912541 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "5199d8d6f3764fdd9ef0fbfcd952889b"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "1f69c42ff20650a1de466baf9cc37d0d"
server_key_iv: "d46489b1be8ee478a9c6f9b8dcb3f8ea"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.912919 19501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "5199d8d6f3764fdd9ef0fbfcd952889b"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "1f69c42ff20650a1de466baf9cc37d0d"
server_key_iv: "d46489b1be8ee478a9c6f9b8dcb3f8ea"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.914474 19501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260811 02:38:11.915460 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.915701 19501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:11.915827 19501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "5199d8d6f3764fdd9ef0fbfcd952889b"
format_stamp: "Formatted at 2026-08-11 02:38:11 on dist-test-slave-cvwc"
server_key: "1f69c42ff20650a1de466baf9cc37d0d"
server_key_iv: "d46489b1be8ee478a9c6f9b8dcb3f8ea"
server_key_version: "encryptionkey@0"
I20260811 02:38:11.915902 19501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:11.919617 19501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:11.919914 19501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:11.920040 19501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:11.920243 19501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:11.920553 19501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:11.920586 19501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.920615 19501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:11.920639 19501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:11.926566 19501 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:33271
I20260811 02:38:11.926611 19717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:33271 every 8 connection(s)
I20260811 02:38:11.926957 19501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:38:11.931255 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.927956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:38193 (local address 127.2.251.62:39171)
0811 02:38:11.928128 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:11.928130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:11.928481 (+ 351us) server_negotiation.cc:407] Connection header received
0811 02:38:11.929159 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:11.929161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:11.929243 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:11.929784 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:11.931001 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:11.931003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:11.931003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:11.931007 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:11.931022 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:11.931027 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:11.931083 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:11.931176 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 02:38:11.931192 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260811 02:38:11.931488 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:11.927951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39171 (local address 127.2.251.1:38193)
0811 02:38:11.928317 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:11.928358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:11.928998 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:11.929868 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:11.929873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:11.929882 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:11.930814 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:38:11.930815 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:11.930847 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:38:11.930879 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:11.931086 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:11.931089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:11.931092 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:38:11.931147 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:38:11.931213 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":216,"threads_started":1}
I20260811 02:38:11.932109 19718 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39171
I20260811 02:38:11.932210 19718 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:11.932399 19718 heartbeater.cc:511] Master 127.2.251.62:39171 requested a full tablet report, sending...
I20260811 02:38:11.933139 19396 ts_manager.cc:194] Registered new tserver with Master: 5199d8d6f3764fdd9ef0fbfcd952889b (127.2.251.1:33271)
I20260811 02:38:11.935243 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 19501
I20260811 02:38:11.935340 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:38:11.935529 3052 external_mini_cluster.cc:1428] Setting key 3543ee05d82c7a8bf46c4185b6e95727
I20260811 02:38:11.936477 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39171
--builtin_ntp_servers=127.2.251.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 02:38:12.029048 19722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:12.029282 19722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:12.029313 19722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:12.030573 19722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:12.030627 19722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:12.030663 19722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:38:12.032171 19722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:12.032456 19722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:12.032809 19722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:12.038656 19732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:12.038653 19729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:12.038793 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:12.038954 19722 server_base.cc:1034] running on GCE node
I20260811 02:38:12.039376 19722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:12.039673 19722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:12.040853 19722 hybrid_clock.cc:648] HybridClock initialized: now 1786415892040823 us; error 54 us; skew 500 ppm
I20260811 02:38:12.042415 19722 webserver.cc:460] Webserver started at http://127.2.251.2:34033/ using document root <none> and password file <none>
I20260811 02:38:12.042668 19722 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:12.042716 19722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:12.042845 19722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:12.044014 19722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "3efcc0de620940fe873a1a2ab0e2595b"
format_stamp: "Formatted at 2026-08-11 02:38:12 on dist-test-slave-cvwc"
server_key: "8a1e065cae79c4f42c58979dfba730f9"
server_key_iv: "0450ef224e764449602643c245576426"
server_key_version: "encryptionkey@0"
I20260811 02:38:12.044354 19722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "3efcc0de620940fe873a1a2ab0e2595b"
format_stamp: "Formatted at 2026-08-11 02:38:12 on dist-test-slave-cvwc"
server_key: "8a1e065cae79c4f42c58979dfba730f9"
server_key_iv: "0450ef224e764449602643c245576426"
server_key_version: "encryptionkey@0"
I20260811 02:38:12.045670 19722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260811 02:38:12.046566 19746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:12.046731 19722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:12.046838 19722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "3efcc0de620940fe873a1a2ab0e2595b"
format_stamp: "Formatted at 2026-08-11 02:38:12 on dist-test-slave-cvwc"
server_key: "8a1e065cae79c4f42c58979dfba730f9"
server_key_iv: "0450ef224e764449602643c245576426"
server_key_version: "encryptionkey@0"
I20260811 02:38:12.046912 19722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:12.050477 19722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:12.050758 19722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:12.050865 19722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:12.051236 19722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:12.051582 19722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:12.051618 19722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:12.051640 19722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:12.051651 19722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:12.057551 19722 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:41255
I20260811 02:38:12.057596 19970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:41255 every 8 connection(s)
I20260811 02:38:12.057986 19722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:38:12.061323 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:12.058748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52151 (local address 127.2.251.62:39171)
0811 02:38:12.058824 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:12.058826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:12.059021 (+ 195us) server_negotiation.cc:407] Connection header received
0811 02:38:12.059564 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:12.059566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:12.059573 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:12.059641 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:12.060989 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:12.060991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:12.060992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:12.060996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:12.061013 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:12.061017 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:12.061079 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:12.061242 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 02:38:12.061258 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 02:38:12.061568 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:12.058698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39171 (local address 127.2.251.2:52151)
0811 02:38:12.058885 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:12.058935 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:12.059435 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:12.059705 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:12.059708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:12.059717 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:12.060781 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:38:12.060782 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:12.060814 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:38:12.060847 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:12.061110 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:12.061117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:12.061121 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:38:12.061211 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 02:38:12.061270 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260811 02:38:12.062170 19971 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39171
I20260811 02:38:12.062286 19971 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:12.062444 19971 heartbeater.cc:511] Master 127.2.251.62:39171 requested a full tablet report, sending...
I20260811 02:38:12.062820 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 19722
I20260811 02:38:12.062849 19396 ts_manager.cc:194] Registered new tserver with Master: 3efcc0de620940fe873a1a2ab0e2595b (127.2.251.2:41255)
I20260811 02:38:12.062907 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:38:12.063117 3052 external_mini_cluster.cc:1428] Setting key a0342c768453eede0672bdb7d18d1ad3
I20260811 02:38:12.064033 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39171
--builtin_ntp_servers=127.2.251.20:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 02:38:12.156396 19977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:12.156633 19977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:12.156656 19977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:12.157883 19977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:12.157934 19977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:12.157963 19977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:38:12.159449 19977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:12.159799 19977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:12.160125 19977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:12.166093 19986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:12.166093 19985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:12.166193 19990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:12.166275 19977 server_base.cc:1034] running on GCE node
I20260811 02:38:12.166638 19977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:12.166937 19977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:12.168176 19977 hybrid_clock.cc:648] HybridClock initialized: now 1786415892168141 us; error 34 us; skew 500 ppm
I20260811 02:38:12.170358 19977 webserver.cc:460] Webserver started at http://127.2.251.3:42859/ using document root <none> and password file <none>
I20260811 02:38:12.170610 19977 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:12.170661 19977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:12.170797 19977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:12.171926 19977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "cce2857bfc4f4429a601c7d79f8e9f03"
format_stamp: "Formatted at 2026-08-11 02:38:12 on dist-test-slave-cvwc"
server_key: "6c8111c333c363987620beeb837d4c4e"
server_key_iv: "2bb7359afe85db919ae3936a45746cb4"
server_key_version: "encryptionkey@0"
I20260811 02:38:12.172308 19977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "cce2857bfc4f4429a601c7d79f8e9f03"
format_stamp: "Formatted at 2026-08-11 02:38:12 on dist-test-slave-cvwc"
server_key: "6c8111c333c363987620beeb837d4c4e"
server_key_iv: "2bb7359afe85db919ae3936a45746cb4"
server_key_version: "encryptionkey@0"
I20260811 02:38:12.173897 19977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260811 02:38:12.174798 20002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:12.175004 19977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:12.175096 19977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "cce2857bfc4f4429a601c7d79f8e9f03"
format_stamp: "Formatted at 2026-08-11 02:38:12 on dist-test-slave-cvwc"
server_key: "6c8111c333c363987620beeb837d4c4e"
server_key_iv: "2bb7359afe85db919ae3936a45746cb4"
server_key_version: "encryptionkey@0"
I20260811 02:38:12.175160 19977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:12.178752 19977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:12.179040 19977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:12.179162 19977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:12.179392 19977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:12.179699 19977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:12.179733 19977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:12.179760 19977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:12.179778 19977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:12.186352 19977 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:39011
I20260811 02:38:12.186416 20192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:39011 every 8 connection(s)
I20260811 02:38:12.186750 19977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:38:12.189877 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:12.187620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:36755 (local address 127.2.251.62:39171)
0811 02:38:12.187667 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:12.187669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:12.187872 (+ 203us) server_negotiation.cc:407] Connection header received
0811 02:38:12.188419 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:12.188420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:12.188426 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:12.188485 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:12.189632 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:12.189633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:12.189634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:38:12.189637 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:12.189650 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:38:12.189654 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:12.189695 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:12.189802 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 02:38:12.189815 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:38:12.190075 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:12.187579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:39171 (local address 127.2.251.3:36755)
0811 02:38:12.187767 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:12.187801 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:12.188293 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:12.188544 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:12.188548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:12.188557 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:12.189492 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:38:12.189492 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:12.189523 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:38:12.189555 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:12.189703 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:12.189706 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:12.189708 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:38:12.189811 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 02:38:12.189867 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260811 02:38:12.190227 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 19977
I20260811 02:38:12.190316 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:38:12.190515 3052 external_mini_cluster.cc:1428] Setting key 46ab3be919e949b25c0a94c1a9576664
I20260811 02:38:12.190541 20193 heartbeater.cc:347] Connected to a master server at 127.2.251.62:39171
I20260811 02:38:12.190644 20193 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:12.190801 20193 heartbeater.cc:511] Master 127.2.251.62:39171 requested a full tablet report, sending...
I20260811 02:38:12.191191 19398 ts_manager.cc:194] Registered new tserver with Master: cce2857bfc4f4429a601c7d79f8e9f03 (127.2.251.3:39011)
I20260811 02:38:12.202735 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 02:38:12.204876 19462 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:38:12.204240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.2.251.62:39171)
0811 02:38:12.204329 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:12.204331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:12.204398 (+ 67us) server_negotiation.cc:407] Connection header received
0811 02:38:12.204475 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:12.204476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:12.204481 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:12.204554 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:12.204791 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52314: BlockingRecv error: recv got EOF from 127.0.0.1:52314 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 02:38:12.204938 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 19501
I20260811 02:38:12.220467 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 19722
I20260811 02:38:12.263216 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 19977
I20260811 02:38:12.278872 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 19334
2026-08-11T02:38:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (700 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:38:12 dist-test-slave-cvwc krb5kdc[20209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:38:12 dist-test-slave-cvwc krb5kdc[20209](info): set up 2 sockets
Aug 11 02:38:12 dist-test-slave-cvwc krb5kdc[20209](info): commencing operation
krb5kdc: starting...
W20260811 02:38:13.507339 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:38:13 dist-test-slave-cvwc krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:38:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:38:13.640982 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:43997
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:38:13.728000 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:13.728231 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:13.728266 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:13.729473 20229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:38:13.729521 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:13.729540 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:13.729557 20229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:38:13.729573 20229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:38:13.731223 20229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:13.731514 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:13.731740 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:13.737387 20239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:13.737385 20238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:13.737756 20229 server_base.cc:1034] running on GCE node
W20260811 02:38:13.737419 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:13.738127 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:13.738440 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:13.739624 20229 hybrid_clock.cc:648] HybridClock initialized: now 1786415893739607 us; error 35 us; skew 500 ppm
Aug 11 02:38:13 dist-test-slave-cvwc krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:38:13.742381 20229 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:38:13.742830 20229 webserver.cc:460] Webserver started at http://127.2.251.62:43151/ using document root <none> and password file <none>
I20260811 02:38:13.743065 20229 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:13.743112 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:13.743229 20229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:13.744326 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "03f46fc432e942b0b244b69fc06485f8"
format_stamp: "Formatted at 2026-08-11 02:38:13 on dist-test-slave-cvwc"
server_key: "a795174ed90b25bef9e68752f3cd489b"
server_key_iv: "09b027bd1961a05f5685467dd696a1b1"
server_key_version: "encryptionkey@0"
I20260811 02:38:13.744668 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "03f46fc432e942b0b244b69fc06485f8"
format_stamp: "Formatted at 2026-08-11 02:38:13 on dist-test-slave-cvwc"
server_key: "a795174ed90b25bef9e68752f3cd489b"
server_key_iv: "09b027bd1961a05f5685467dd696a1b1"
server_key_version: "encryptionkey@0"
I20260811 02:38:13.746362 20229 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:38:13.747259 20263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:13.747628 20229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:13.747725 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "03f46fc432e942b0b244b69fc06485f8"
format_stamp: "Formatted at 2026-08-11 02:38:13 on dist-test-slave-cvwc"
server_key: "a795174ed90b25bef9e68752f3cd489b"
server_key_iv: "09b027bd1961a05f5685467dd696a1b1"
server_key_version: "encryptionkey@0"
I20260811 02:38:13.747813 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:13.763420 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:13.765693 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:13.765921 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:13.770068 20229 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:43997
I20260811 02:38:13.770109 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:43997 every 8 connection(s)
I20260811 02:38:13.770533 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:38:13.771888 20349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:38:13.774405 20349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8: Bootstrap starting.
I20260811 02:38:13.775043 20349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8: Neither blocks nor log segments found. Creating new log.
I20260811 02:38:13.775332 20349 log.cc:824] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8: Log is configured to *not* fsync() on all Append() calls
I20260811 02:38:13.776154 20349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8: No bootstrap required, opened a new log
I20260811 02:38:13.777619 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 20229
I20260811 02:38:13.777702 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:38:13.777918 3052 external_mini_cluster.cc:1428] Setting key 8dbf3d64f3210f94d3ccad78d9e762b1
I20260811 02:38:13.777891 20349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f46fc432e942b0b244b69fc06485f8" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 43997 } }
I20260811 02:38:13.778055 20349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:38:13.778100 20349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f46fc432e942b0b244b69fc06485f8, State: Initialized, Role: FOLLOWER
I20260811 02:38:13.778249 20349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f46fc432e942b0b244b69fc06485f8" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 43997 } }
I20260811 02:38:13.778329 20349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:38:13.778362 20349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:38:13.778401 20349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:38:13.779050 20349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f46fc432e942b0b244b69fc06485f8" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 43997 } }
I20260811 02:38:13.779152 20349 leader_election.cc:302] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03f46fc432e942b0b244b69fc06485f8; no voters:
I20260811 02:38:13.779325 20349 leader_election.cc:288] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:38:13.779356 20361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:38:13.779500 20361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [term 1 LEADER]: Becoming Leader. State: Replica: 03f46fc432e942b0b244b69fc06485f8, State: Running, Role: LEADER
I20260811 02:38:13.779577 20349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:38:13.779593 20361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f46fc432e942b0b244b69fc06485f8" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 43997 } }
I20260811 02:38:13.780061 20362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03f46fc432e942b0b244b69fc06485f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f46fc432e942b0b244b69fc06485f8" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 43997 } } }
I20260811 02:38:13.780139 20362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [sys.catalog]: This master's current role is: LEADER
I20260811 02:38:13.780517 20373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:38:13.781085 20373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:38:13.781061 20363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03f46fc432e942b0b244b69fc06485f8. Latest consensus state: current_term: 1 leader_uuid: "03f46fc432e942b0b244b69fc06485f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f46fc432e942b0b244b69fc06485f8" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 43997 } } }
I20260811 02:38:13.781344 20363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8 [sys.catalog]: This master's current role is: LEADER
I20260811 02:38:13.782822 20373 catalog_manager.cc:1347] Generated new cluster ID: ca3e66114a84428f893c14ab2e442b41
I20260811 02:38:13.782866 20373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:38:13 dist-test-slave-cvwc krb5kdc[20209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:38:13.789651 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:13.778386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.2.251.62:43997)
0811 02:38:13.778640 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:13.778651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:13.778677 (+ 26us) server_negotiation.cc:407] Connection header received
0811 02:38:13.779005 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:13.779011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:13.779062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:13.779186 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:13.779946 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:13.781415 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:13.782290 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:13.782885 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:13.782993 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:13.785885 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:13.785889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:13.785890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:38:13.785902 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:13.787897 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:13.788318 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:13.788322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:13.788322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:13.788468 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:13.788649 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:13.788650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:13.788650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:13.788991 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:13.789164 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:13.789242 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 02:38:13.789416 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260811 02:38:13.792160 20373 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:38:13.792640 20373 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:38:13.805822 20373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8: Generated new TSK 0
I20260811 02:38:13.806030 20373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:38:13.841859 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:43997
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:38:13.935071 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:13.935310 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:13.935338 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:13.936956 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:13.937011 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:13.937039 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:38:13.938753 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:13.939000 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:13.939239 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:13.945230 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:13.945307 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:13.945480 20393 server_base.cc:1034] running on GCE node
W20260811 02:38:13.945307 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:13.945772 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:13.946017 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:13.947186 20393 hybrid_clock.cc:648] HybridClock initialized: now 1786415893947172 us; error 31 us; skew 500 ppm
Aug 11 02:38:13 dist-test-slave-cvwc krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:38:13.950261 20393 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:38:13.951102 20393 webserver.cc:460] Webserver started at http://127.2.251.1:45341/ using document root <none> and password file <none>
I20260811 02:38:13.951416 20393 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:13.951475 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:13.951774 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:13.953012 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "ee966641642a4aa29f6ed93d5d1e7c8f"
format_stamp: "Formatted at 2026-08-11 02:38:13 on dist-test-slave-cvwc"
server_key: "f34fefddb812e57cbb33def091d90083"
server_key_iv: "c12d77aae8df11249e12d6c1637d3163"
server_key_version: "encryptionkey@0"
I20260811 02:38:13.953377 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "ee966641642a4aa29f6ed93d5d1e7c8f"
format_stamp: "Formatted at 2026-08-11 02:38:13 on dist-test-slave-cvwc"
server_key: "f34fefddb812e57cbb33def091d90083"
server_key_iv: "c12d77aae8df11249e12d6c1637d3163"
server_key_version: "encryptionkey@0"
I20260811 02:38:13.954910 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260811 02:38:13.955819 20423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:13.956125 20393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:38:13.956233 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "ee966641642a4aa29f6ed93d5d1e7c8f"
format_stamp: "Formatted at 2026-08-11 02:38:13 on dist-test-slave-cvwc"
server_key: "f34fefddb812e57cbb33def091d90083"
server_key_iv: "c12d77aae8df11249e12d6c1637d3163"
server_key_version: "encryptionkey@0"
I20260811 02:38:13.956312 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:13.967242 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:13.970069 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:13.970247 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:13.970527 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:13.970867 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:13.970903 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:13.970935 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:13.970958 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:13.976812 20393 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:40005
I20260811 02:38:13.976887 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:40005 every 8 connection(s)
I20260811 02:38:13.977209 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:38:13.978880 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 20393
I20260811 02:38:13.979030 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:38:13.979207 3052 external_mini_cluster.cc:1428] Setting key d965c5f79238cf569119f4dabbf32aa9
Aug 11 02:38:13 dist-test-slave-cvwc krb5kdc[20209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415893, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:38:13.987802 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:13.978172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:47451 (local address 127.2.251.62:43997)
0811 02:38:13.978248 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:13.978250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:13.978297 (+ 47us) server_negotiation.cc:407] Connection header received
0811 02:38:13.978892 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:13.978895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:13.978919 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:13.979020 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:13.980225 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:13.980945 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:13.982105 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:13.982405 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:13.982484 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:13.984989 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:13.984991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:13.984991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:38:13.985001 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:13.986374 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:13.986823 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:13.986824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:13.986824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:13.986879 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:13.987114 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:13.987115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:13.987116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:13.987237 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:13.987295 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:13.987597 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 02:38:13.987697 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:38:13.987982 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:13.977999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:43997 (local address 127.2.251.1:47451)
0811 02:38:13.978169 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:13.978213 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:13.978742 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:13.979157 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:13.979162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:13.979919 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:13.980130 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:13.980136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:13.981034 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:13.981035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:13.982013 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:13.982016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:13.982503 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:13.982504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:13.982617 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:13.983130 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:13.983140 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:13.984879 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:13.986464 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:38:13.986474 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:38:13.986478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:38:13.986722 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:38:13.986940 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:38:13.986940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:38:13.986941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:38:13.987056 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:38:13.987315 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:13.987322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:13.987469 (+ 147us) client_negotiation.cc:769] Sending connection context
0811 02:38:13.987567 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 02:38:13.987745 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:38:13.988536 20626 heartbeater.cc:347] Connected to a master server at 127.2.251.62:43997
I20260811 02:38:13.988651 20626 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:13.988850 20626 heartbeater.cc:511] Master 127.2.251.62:43997 requested a full tablet report, sending...
I20260811 02:38:13.989475 20290 ts_manager.cc:194] Registered new tserver with Master: ee966641642a4aa29f6ed93d5d1e7c8f (127.2.251.1:40005)
I20260811 02:38:13.990164 20290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:47451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:38:14.010142 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:43997
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:38:14.099828 20635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:14.100077 20635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:14.100114 20635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:14.101369 20635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:14.101423 20635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:14.101461 20635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:38:14.103092 20635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:14.103374 20635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:14.103619 20635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:14.109549 20649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:14.109557 20646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:14.109822 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:14.110002 20635 server_base.cc:1034] running on GCE node
I20260811 02:38:14.110231 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:14.110486 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:14.111743 20635 hybrid_clock.cc:648] HybridClock initialized: now 1786415894111722 us; error 43 us; skew 500 ppm
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415894, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:38:14.114888 20635 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:38:14.115394 20635 webserver.cc:460] Webserver started at http://127.2.251.2:37967/ using document root <none> and password file <none>
I20260811 02:38:14.115702 20635 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:14.115774 20635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:14.115903 20635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:14.117003 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "d298570b816c4ea3bde1840211a209ea"
format_stamp: "Formatted at 2026-08-11 02:38:14 on dist-test-slave-cvwc"
server_key: "ff30335a3a3aa374706e3af72262002e"
server_key_iv: "ddfd644b863f4e9fa3b4edcc4e90bcfc"
server_key_version: "encryptionkey@0"
I20260811 02:38:14.117350 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "d298570b816c4ea3bde1840211a209ea"
format_stamp: "Formatted at 2026-08-11 02:38:14 on dist-test-slave-cvwc"
server_key: "ff30335a3a3aa374706e3af72262002e"
server_key_iv: "ddfd644b863f4e9fa3b4edcc4e90bcfc"
server_key_version: "encryptionkey@0"
I20260811 02:38:14.118758 20635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:38:14.119618 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:14.119900 20635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:14.120007 20635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "d298570b816c4ea3bde1840211a209ea"
format_stamp: "Formatted at 2026-08-11 02:38:14 on dist-test-slave-cvwc"
server_key: "ff30335a3a3aa374706e3af72262002e"
server_key_iv: "ddfd644b863f4e9fa3b4edcc4e90bcfc"
server_key_version: "encryptionkey@0"
I20260811 02:38:14.120173 20635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:14.142251 20635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:14.145315 20635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:14.145481 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:14.145773 20635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:14.146121 20635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:14.146157 20635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:14.146186 20635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:14.146204 20635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:14.152320 20635 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:45667
I20260811 02:38:14.152369 20859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:45667 every 8 connection(s)
I20260811 02:38:14.152694 20635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:38:14.157248 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 20635
I20260811 02:38:14.157369 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:38:14.157550 3052 external_mini_cluster.cc:1428] Setting key d51a19701010895e5a4410dd08482a04
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[20209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415894, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:38:14.165284 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.153526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52267 (local address 127.2.251.62:43997)
0811 02:38:14.153666 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.153677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.153950 (+ 273us) server_negotiation.cc:407] Connection header received
0811 02:38:14.154704 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.154705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.154721 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.154781 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:14.156027 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.156708 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.158205 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.158522 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.158603 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.161481 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:14.161483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:14.161484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:38:14.161492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:14.163338 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:14.163979 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:14.163981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:14.163981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:14.164079 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:14.164344 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:14.164345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:14.164346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:14.164495 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:14.164565 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.164926 (+ 361us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.165172 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 02:38:14.165648 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.153418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:43997 (local address 127.2.251.2:52267)
0811 02:38:14.153691 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.153826 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.154563 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.154884 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.154889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.155649 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:14.155914 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.155920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.156911 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.156913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.158069 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.158074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.158620 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.158621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.158749 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.159364 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:14.159375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:14.161328 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:14.163482 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:38:14.163495 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:38:14.163498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:38:14.163896 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:38:14.164149 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:38:14.164150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:38:14.164150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:38:14.164280 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:38:14.164586 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:14.164595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:14.164779 (+ 184us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.164887 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.165297 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260811 02:38:14.166181 20860 heartbeater.cc:347] Connected to a master server at 127.2.251.62:43997
I20260811 02:38:14.166294 20860 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:14.166510 20860 heartbeater.cc:511] Master 127.2.251.62:43997 requested a full tablet report, sending...
I20260811 02:38:14.166913 20290 ts_manager.cc:194] Registered new tserver with Master: d298570b816c4ea3bde1840211a209ea (127.2.251.2:45667)
I20260811 02:38:14.167316 20290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:52267
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:38:14.190212 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:43997
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:38:14.277411 20867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:38:14.277642 20867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:38:14.277668 20867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:38:14.278903 20867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:38:14.278954 20867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:38:14.278981 20867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:38:14.280689 20867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:38:14.281116 20867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:38:14.281510 20867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:38:14.286717 20875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:14.286728 20877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:38:14.286792 20880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:38:14.287087 20867 server_base.cc:1034] running on GCE node
I20260811 02:38:14.287312 20867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:38:14.287644 20867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:38:14.288829 20867 hybrid_clock.cc:648] HybridClock initialized: now 1786415894288807 us; error 31 us; skew 500 ppm
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[20209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415894, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:38:14.291767 20867 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:38:14.292281 20867 webserver.cc:460] Webserver started at http://127.2.251.3:45425/ using document root <none> and password file <none>
I20260811 02:38:14.292519 20867 fs_manager.cc:359] Metadata directory not provided
I20260811 02:38:14.292570 20867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:38:14.292703 20867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:38:14.293862 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "eab4e732f49b42cd88cfc87bf7940353"
format_stamp: "Formatted at 2026-08-11 02:38:14 on dist-test-slave-cvwc"
server_key: "8c74d7baeb4cd86082b5273b527e95d5"
server_key_iv: "585182a035d1dba331dae9545543ebb3"
server_key_version: "encryptionkey@0"
I20260811 02:38:14.294221 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "eab4e732f49b42cd88cfc87bf7940353"
format_stamp: "Formatted at 2026-08-11 02:38:14 on dist-test-slave-cvwc"
server_key: "8c74d7baeb4cd86082b5273b527e95d5"
server_key_iv: "585182a035d1dba331dae9545543ebb3"
server_key_version: "encryptionkey@0"
I20260811 02:38:14.295545 20867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:38:14.296473 20892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:14.296705 20867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:38:14.296802 20867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "eab4e732f49b42cd88cfc87bf7940353"
format_stamp: "Formatted at 2026-08-11 02:38:14 on dist-test-slave-cvwc"
server_key: "8c74d7baeb4cd86082b5273b527e95d5"
server_key_iv: "585182a035d1dba331dae9545543ebb3"
server_key_version: "encryptionkey@0"
I20260811 02:38:14.296875 20867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:38:14.310034 20867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:38:14.312336 20867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:38:14.312505 20867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:38:14.312762 20867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:38:14.313109 20867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:38:14.313148 20867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:14.313177 20867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:38:14.313202 20867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:38:14.319456 20867 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:45377
I20260811 02:38:14.319502 21108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:45377 every 8 connection(s)
I20260811 02:38:14.319839 20867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[20209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415894, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:38:14.326267 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 20867
I20260811 02:38:14.326345 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:38:14.326541 3052 external_mini_cluster.cc:1428] Setting key a65efd90c166f24aa89f0d117854bfff
I20260811 02:38:14.329429 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.320608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:57693 (local address 127.2.251.62:43997)
0811 02:38:14.320670 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.320672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.320838 (+ 166us) server_negotiation.cc:407] Connection header received
0811 02:38:14.321407 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.321408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.321422 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.321475 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:14.322532 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.323033 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.324049 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.324234 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.324290 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.326531 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:14.326533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:14.326533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:38:14.326541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:14.327990 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:14.328435 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:14.328436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:14.328436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:14.328492 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:14.328783 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:14.328783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:14.328783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:14.328910 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:14.328961 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.329286 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.329356 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:38:14.329597 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.320529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:43997 (local address 127.2.251.3:57693)
0811 02:38:14.320705 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.320754 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.321295 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.321551 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.321555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.322223 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:38:14.322434 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.322440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.323119 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.323120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.323967 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.323970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.324306 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.324307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.324426 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.324926 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:38:14.324935 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:38:14.326441 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:38:14.328077 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:38:14.328085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:38:14.328087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:38:14.328344 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:38:14.328585 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:38:14.328586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:38:14.328586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:38:14.328721 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:38:14.328991 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:38:14.328999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:38:14.329142 (+ 143us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.329240 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.329381 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260811 02:38:14.330157 21109 heartbeater.cc:347] Connected to a master server at 127.2.251.62:43997
I20260811 02:38:14.330262 21109 heartbeater.cc:464] Registering TS with master...
I20260811 02:38:14.330513 21109 heartbeater.cc:511] Master 127.2.251.62:43997 requested a full tablet report, sending...
I20260811 02:38:14.330909 20290 ts_manager.cc:194] Registered new tserver with Master: eab4e732f49b42cd88cfc87bf7940353 (127.2.251.3:45377)
I20260811 02:38:14.331277 20290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:57693
I20260811 02:38:14.338725 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:38:14.346591 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.340320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.2.251.62:43997)
0811 02:38:14.340373 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.340374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.340453 (+ 79us) server_negotiation.cc:407] Connection header received
0811 02:38:14.340504 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.340505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.340514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.340568 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:38:14.341145 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.341783 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.342532 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.342718 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.342779 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.344024 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:38:14.344025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:38:14.344025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:38:14.344032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:38:14.345319 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:14.345636 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:14.345636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:14.345637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:14.345684 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:38:14.346062 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:38:14.346063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:38:14.346063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:38:14.346247 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:38:14.346335 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.346450 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.346522 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:38:14.348513 20290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:38:14.350240 20290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:38:14.356834 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.352618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:45667 (local address 127.0.0.1:41122)
0811 02:38:14.352950 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.352993 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.353067 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.353279 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.353282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.353284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:38:14.353364 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.353366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.354756 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.354757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.355853 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.355855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.356504 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.356505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.356592 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.356607 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.356675 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.356734 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":32,"thread_start_us":67,"threads_started":1}
I20260811 02:38:14.356837 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.352877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40005 (local address 127.0.0.1:35038)
0811 02:38:14.353218 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.353228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.353282 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.353525 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.353526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.353528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:38:14.353584 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.353586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.354506 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.354507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.355778 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.355781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.356648 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.356649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.356701 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.356705 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.356749 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.356781 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260811 02:38:14.356904 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.352981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.2.251.1:40005)
0811 02:38:14.353165 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.353169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.353333 (+ 164us) server_negotiation.cc:407] Connection header received
0811 02:38:14.353373 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.353375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.353389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.353452 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:38:14.353661 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.354408 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.355866 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.356576 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.356635 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.356685 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.356763 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.356804 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260811 02:38:14.356904 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.352731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.2.251.3:45377)
0811 02:38:14.353006 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.353011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.353068 (+ 57us) server_negotiation.cc:407] Connection header received
0811 02:38:14.353193 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.353195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.353210 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.353273 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:38:14.353479 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.354658 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.355707 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.356434 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.356503 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.356569 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.356763 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.356809 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20260811 02:38:14.357126 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.352767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.2.251.2:45667)
0811 02:38:14.353097 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.353101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.353112 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:38:14.353156 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.353158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.353176 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.353270 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:38:14.353470 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.354647 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.355930 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.356424 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.356490 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.356560 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.356995 (+ 435us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.357045 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":80,"threads_started":1}
I20260811 02:38:14.357326 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.352629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:45377 (local address 127.0.0.1:55658)
0811 02:38:14.352993 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.353008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.353094 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.353370 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.353371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.353372 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:38:14.353403 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.353404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.354772 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.354773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.355643 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.355646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.356606 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.356607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.356671 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.356675 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.357209 (+ 534us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.357260 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":107,"threads_started":1}
I20260811 02:38:14.358040 20983 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd1ba14f7f40acb579dd89140be4f0 (DEFAULT_TABLE table=test-table [id=39c2d2e4f8b64fed9396ef5619c8a9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:38:14.358035 20509 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd1ba14f7f40acb579dd89140be4f0 (DEFAULT_TABLE table=test-table [id=39c2d2e4f8b64fed9396ef5619c8a9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:38:14.358153 20737 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd1ba14f7f40acb579dd89140be4f0 (DEFAULT_TABLE table=test-table [id=39c2d2e4f8b64fed9396ef5619c8a9a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:38:14.358441 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dd1ba14f7f40acb579dd89140be4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:38:14.358441 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dd1ba14f7f40acb579dd89140be4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:38:14.358505 20737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78dd1ba14f7f40acb579dd89140be4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:38:14.361577 21142 tablet_bootstrap.cc:492] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Bootstrap starting.
I20260811 02:38:14.361855 21143 tablet_bootstrap.cc:492] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Bootstrap starting.
I20260811 02:38:14.362151 21145 tablet_bootstrap.cc:492] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: Bootstrap starting.
I20260811 02:38:14.362506 21142 tablet_bootstrap.cc:654] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Neither blocks nor log segments found. Creating new log.
I20260811 02:38:14.362504 21143 tablet_bootstrap.cc:654] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Neither blocks nor log segments found. Creating new log.
I20260811 02:38:14.362739 21145 tablet_bootstrap.cc:654] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: Neither blocks nor log segments found. Creating new log.
I20260811 02:38:14.363112 21142 log.cc:824] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Log is configured to *not* fsync() on all Append() calls
I20260811 02:38:14.363112 21143 log.cc:824] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Log is configured to *not* fsync() on all Append() calls
I20260811 02:38:14.363194 21145 log.cc:824] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: Log is configured to *not* fsync() on all Append() calls
I20260811 02:38:14.363948 21142 tablet_bootstrap.cc:492] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: No bootstrap required, opened a new log
I20260811 02:38:14.363962 21143 tablet_bootstrap.cc:492] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: No bootstrap required, opened a new log
I20260811 02:38:14.364012 21143 ts_tablet_manager.cc:1397] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 02:38:14.364015 21142 ts_tablet_manager.cc:1397] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 02:38:14.365428 21143 raft_consensus.cc:348] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.365475 21142 raft_consensus.cc:348] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.365579 21143 raft_consensus.cc:374] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:38:14.365597 21142 raft_consensus.cc:374] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:38:14.365603 21143 raft_consensus.cc:729] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab4e732f49b42cd88cfc87bf7940353, State: Initialized, Role: FOLLOWER
I20260811 02:38:14.365628 21142 raft_consensus.cc:729] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d298570b816c4ea3bde1840211a209ea, State: Initialized, Role: FOLLOWER
I20260811 02:38:14.365756 21143 consensus_queue.cc:260] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.365770 21142 consensus_queue.cc:260] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.366051 21143 ts_tablet_manager.cc:1428] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:38:14.366055 21142 ts_tablet_manager.cc:1428] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 02:38:14.366200 20860 heartbeater.cc:503] Master 127.2.251.62:43997 was elected leader, sending a full tablet report...
I20260811 02:38:14.366228 21109 heartbeater.cc:503] Master 127.2.251.62:43997 was elected leader, sending a full tablet report...
I20260811 02:38:14.367327 21145 tablet_bootstrap.cc:492] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: No bootstrap required, opened a new log
I20260811 02:38:14.367422 21145 ts_tablet_manager.cc:1397] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260811 02:38:14.368884 21145 raft_consensus.cc:348] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.369028 21145 raft_consensus.cc:374] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:38:14.369059 21145 raft_consensus.cc:729] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee966641642a4aa29f6ed93d5d1e7c8f, State: Initialized, Role: FOLLOWER
I20260811 02:38:14.369185 21145 consensus_queue.cc:260] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.369465 21145 ts_tablet_manager.cc:1428] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:38:14.369496 20626 heartbeater.cc:503] Master 127.2.251.62:43997 was elected leader, sending a full tablet report...
W20260811 02:38:14.403299 20861 tablet.cc:2369] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:38:14.438862 21155 raft_consensus.cc:482] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:38:14.439050 21155 raft_consensus.cc:504] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.439463 21155 leader_election.cc:288] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eab4e732f49b42cd88cfc87bf7940353 (127.2.251.3:45377), ee966641642a4aa29f6ed93d5d1e7c8f (127.2.251.1:40005)
I20260811 02:38:14.443477 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.439889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:36397 (local address 127.2.251.1:40005)
0811 02:38:14.439944 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.439946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.440092 (+ 146us) server_negotiation.cc:407] Connection header received
0811 02:38:14.440155 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.440157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.440169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.440232 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:38:14.440408 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.441072 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.442439 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.443040 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.443122 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.443133 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.443353 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.443417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 02:38:14.443476 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.439845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:57209 (local address 127.2.251.3:45377)
0811 02:38:14.439917 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.439919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.439929 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:38:14.439961 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.439961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.439973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.440056 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:38:14.440215 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.440944 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.442439 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.443043 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.443122 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.443133 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.443352 (+ 219us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.443399 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 02:38:14.443500 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.439694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:45377 (local address 127.2.251.2:57209)
0811 02:38:14.439743 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.439755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.439805 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.440062 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.440063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.440065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:38:14.440146 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.440148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.441047 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.441048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.442321 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.442324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.443160 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.443161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.443265 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.443268 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.443338 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.443405 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":49}
I20260811 02:38:14.443513 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.439825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40005 (local address 127.2.251.2:36397)
0811 02:38:14.440020 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.440031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:38:14.440089 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:38:14.440284 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:38:14.440285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:38:14.440287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:38:14.440334 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.440336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.441149 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.441149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.442321 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:38:14.442325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.443160 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.443161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:38:14.443253 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.443259 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:38:14.443338 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 02:38:14.443398 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":3,"thread_start_us":141,"threads_started":1}
I20260811 02:38:14.443970 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd1ba14f7f40acb579dd89140be4f0" candidate_uuid: "d298570b816c4ea3bde1840211a209ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" is_pre_election: true
I20260811 02:38:14.443990 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd1ba14f7f40acb579dd89140be4f0" candidate_uuid: "d298570b816c4ea3bde1840211a209ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab4e732f49b42cd88cfc87bf7940353" is_pre_election: true
I20260811 02:38:14.444154 21021 raft_consensus.cc:2454] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d298570b816c4ea3bde1840211a209ea in term 0.
I20260811 02:38:14.444154 20545 raft_consensus.cc:2454] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d298570b816c4ea3bde1840211a209ea in term 0.
I20260811 02:38:14.444412 20659 leader_election.cc:302] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d298570b816c4ea3bde1840211a209ea, ee966641642a4aa29f6ed93d5d1e7c8f; no voters:
I20260811 02:38:14.444577 21155 raft_consensus.cc:2781] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:38:14.444643 21155 raft_consensus.cc:482] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:38:14.444670 21155 raft_consensus.cc:3037] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:38:14.445459 21155 raft_consensus.cc:504] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.445605 21155 leader_election.cc:288] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [CANDIDATE]: Term 1 election: Requested vote from peers eab4e732f49b42cd88cfc87bf7940353 (127.2.251.3:45377), ee966641642a4aa29f6ed93d5d1e7c8f (127.2.251.1:40005)
I20260811 02:38:14.445824 21021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd1ba14f7f40acb579dd89140be4f0" candidate_uuid: "d298570b816c4ea3bde1840211a209ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab4e732f49b42cd88cfc87bf7940353"
I20260811 02:38:14.445844 20545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd1ba14f7f40acb579dd89140be4f0" candidate_uuid: "d298570b816c4ea3bde1840211a209ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f"
I20260811 02:38:14.445914 21021 raft_consensus.cc:3037] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:38:14.445920 20545 raft_consensus.cc:3037] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:38:14.446671 20545 raft_consensus.cc:2454] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d298570b816c4ea3bde1840211a209ea in term 1.
I20260811 02:38:14.446864 21021 raft_consensus.cc:2454] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d298570b816c4ea3bde1840211a209ea in term 1.
I20260811 02:38:14.446884 20659 leader_election.cc:302] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d298570b816c4ea3bde1840211a209ea, ee966641642a4aa29f6ed93d5d1e7c8f; no voters:
I20260811 02:38:14.446981 21155 raft_consensus.cc:2781] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:38:14.447155 21155 raft_consensus.cc:686] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 1 LEADER]: Becoming Leader. State: Replica: d298570b816c4ea3bde1840211a209ea, State: Running, Role: LEADER
I20260811 02:38:14.447247 21155 consensus_queue.cc:237] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } }
I20260811 02:38:14.448169 20288 catalog_manager.cc:5495] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea reported cstate change: term changed from 0 to 1, leader changed from <none> to d298570b816c4ea3bde1840211a209ea (127.2.251.2). New cstate: current_term: 1 leader_uuid: "d298570b816c4ea3bde1840211a209ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d298570b816c4ea3bde1840211a209ea" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 45667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 } health_report { overall_health: UNKNOWN } } }
W20260811 02:38:14.477924 20627 tablet.cc:2369] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:38:14.478085 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.475484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.2.251.2:45667)
0811 02:38:14.475566 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 02:38:14.475568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:38:14.475577 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:38:14.475617 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:38:14.475617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:38:14.475638 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:38:14.475712 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:38:14.475828 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.476338 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.477124 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:38:14.477330 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:38:14.477405 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:38:14.477640 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:38:14.477795 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:38:14.477868 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:38:14.477951 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 02:38:14.477975 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 02:38:14.480093 20545 raft_consensus.cc:1261] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f [term 1 FOLLOWER]: Refusing update from remote peer d298570b816c4ea3bde1840211a209ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:38:14.480093 21021 raft_consensus.cc:1261] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 1 FOLLOWER]: Refusing update from remote peer d298570b816c4ea3bde1840211a209ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:38:14.480451 21155 consensus_queue.cc:1035] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab4e732f49b42cd88cfc87bf7940353" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:38:14.480602 21161 consensus_queue.cc:1035] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee966641642a4aa29f6ed93d5d1e7c8f" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:38:14.483848 21165 mvcc.cc:205] Tried to move back new op lower bound from 7317159503787569152 to 7317159503656607744. Current Snapshot: MvccSnapshot[applied={T|T < 7317159503787569152}]
I20260811 02:38:14.486176 20288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:38:14.486269 20288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:38:14.487052 20288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03f46fc432e942b0b244b69fc06485f8: Sending DeleteTablet for 3 replicas of tablet 78dd1ba14f7f40acb579dd89140be4f0
I20260811 02:38:14.487447 20737 tablet_service.cc:1514] Processing DeleteTablet for tablet 78dd1ba14f7f40acb579dd89140be4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:38:14 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:41122
I20260811 02:38:14.487465 20983 tablet_service.cc:1514] Processing DeleteTablet for tablet 78dd1ba14f7f40acb579dd89140be4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:38:14 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:55658
I20260811 02:38:14.487694 21183 tablet_replica.cc:330] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: stopping tablet replica
I20260811 02:38:14.487708 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet 78dd1ba14f7f40acb579dd89140be4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:38:14 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:35038
I20260811 02:38:14.487811 21183 raft_consensus.cc:2229] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:38:14.487953 21184 tablet_replica.cc:330] T 78dd1ba14f7f40acb579dd89140be4f0 P ee966641642a4aa29f6ed93d5d1e7c8f: stopping tablet replica
I20260811 02:38:14.487960 21183 raft_consensus.cc:2258] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:38:14.488129 21181 tablet_replica.cc:330] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: stopping tablet replica
I20260811 02:38:14.488246 21181 raft_consensus.cc:2229] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:38:14.488358 21181 raft_consensus.cc:2258] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:38:14.488526 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 20393
I20260811 02:38:14.502476 21183 ts_tablet_manager.cc:1905] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:38:14.502630 21181 ts_tablet_manager.cc:1905] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:38:14.504904 21183 ts_tablet_manager.cc:1918] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:38:14.505178 21183 log.cc:1196] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/78dd1ba14f7f40acb579dd89140be4f0
I20260811 02:38:14.505911 21183 ts_tablet_manager.cc:1939] T 78dd1ba14f7f40acb579dd89140be4f0 P d298570b816c4ea3bde1840211a209ea: Deleting consensus metadata
I20260811 02:38:14.506048 21181 ts_tablet_manager.cc:1918] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:38:14.506172 21181 log.cc:1196] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/78dd1ba14f7f40acb579dd89140be4f0
I20260811 02:38:14.506494 21181 ts_tablet_manager.cc:1939] T 78dd1ba14f7f40acb579dd89140be4f0 P eab4e732f49b42cd88cfc87bf7940353: Deleting consensus metadata
I20260811 02:38:14.506930 20271 catalog_manager.cc:4841] TS d298570b816c4ea3bde1840211a209ea (127.2.251.2:45667): tablet 78dd1ba14f7f40acb579dd89140be4f0 (table test-table [id=39c2d2e4f8b64fed9396ef5619c8a9a8]) successfully deleted
I20260811 02:38:14.507001 20265 catalog_manager.cc:4841] TS eab4e732f49b42cd88cfc87bf7940353 (127.2.251.3:45377): tablet 78dd1ba14f7f40acb579dd89140be4f0 (table test-table [id=39c2d2e4f8b64fed9396ef5619c8a9a8]) successfully deleted
W20260811 02:38:14.512111 20265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:40005 (error 108)
I20260811 02:38:14.512487 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 20635
I20260811 02:38:14.549438 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:38:14.548258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40005 (local address 127.0.0.1:35050)
0811 02:38:14.549178 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0811 02:38:14.549383 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:40005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":884}
W20260811 02:38:14.549634 20265 catalog_manager.cc:4584] TS ee966641642a4aa29f6ed93d5d1e7c8f (127.2.251.1:40005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78dd1ba14f7f40acb579dd89140be4f0: Network error: Client connection negotiation failed: client connection to 127.2.251.1:40005: connect: Connection refused (error 111)
I20260811 02:38:14.558560 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 20867
I20260811 02:38:14.579840 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 20229
2026-08-11T02:38:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2309 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[21195](info): set up 2 sockets
Aug 11 02:38:14 dist-test-slave-cvwc krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260811 02:38:15.849844 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:38:15 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:38:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/rangeradmin_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/rangeradmin_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/rangerlookup_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/rangerlookup_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
I20260811 02:38:16.057806 3052 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRZ17kF/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/postgres -l logfile start
2026-08-11 02:38:19.084 UTC [21253] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 02:38:19.084 UTC [21253] LOG: listening on IPv4 address "127.2.251.20", port 49433
2026-08-11 02:38:19.088 UTC [21253] LOG: listening on Unix socket "/tmp/.s.PGSQL.49433"
2026-08-11 02:38:19.118 UTC [21261] LOG: database system was shut down at 2026-08-11 02:38:17 UTC
2026-08-11 02:38:19.123 UTC [21253] LOG: database system is ready to accept connections
I20260811 02:38:20.234277 3052 mini_postgres.cc:96] Postgres bound to 49433
2026-08-11 02:38:20.243 UTC [21270] FATAL: database "slave" does not exist
127.2.251.20:49433 - accepting connections
I20260811 02:38:20.244939 3052 mini_ranger.cc:161] Starting Ranger...
I20260811 02:38:20.264765 3052 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 02:38:20.823325 3052 mini_ranger.cc:87] Created ranger Postgres database
I20260811 02:38:20.823630 3052 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-admin
2026-08-11 02:38:21,190 [I] DB FLAVOR :POSTGRES
2026-08-11 02:38:21,191 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 02:38:21,191 [I] Checking connection..
2026-08-11 02:38:21,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 02:38:21,486 [I] Checking connection passed.
2026-08-11 02:38:21,487 [I] --------- Verifying version history table ---------
2026-08-11 02:38:21,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 02:38:21,787 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 02:38:21,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 02:38:22,077 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 02:38:22,077 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 02:38:22,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 02:38:22.350 UTC [21371] WARNING: there is no transaction in progress
2026-08-11 02:38:22,360 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 02:38:22,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 02:38:22,614 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 02:38:22,614 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 02:38:22,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 02:38:22,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 02:38:23,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cvwc.c.gcp-upstream.internal','N') ;"
2026-08-11 02:38:23,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 02:38:23,752 [I] Table x_portal_user does not exist in database ranger
2026-08-11 02:38:23,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 02:38:24,027 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 02:38:24,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 02:38:24,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 02:38:24,578 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 02:38:24,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 02:38:25.678 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.696 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.715 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.733 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.751 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.770 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.845 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.853 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.871 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.884 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.895 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.911 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.921 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.928 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.936 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:25.943 UTC [21574] WARNING: there is no transaction in progress
2026-08-11 02:38:26,757 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 02:38:26,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 02:38:27,021 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 02:38:27,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 02:38:27,284 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 02:38:27,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 02:38:27,575 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 02:38:27,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 02:38:27,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 02:38:28,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cvwc.c.gcp-upstream.internal';"
2026-08-11 02:38:28,459 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 02:38:28,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 02:38:28,769 [I] DB_PATCHES have already been applied
I20260811 02:38:28.775451 3052 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRZ17kF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 02:38:28.775533 3052 mini_ranger.cc:193] Using host: 127.2.251.20
I20260811 02:38:28.779785 3052 mini_ranger.cc:227] Ranger admin URL: http://127.2.251.20:39893
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 2:38:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 02:38:30.113538 3052 mini_ranger.cc:160] Time spent starting Ranger: real 9.869s user 0.001s sys 0.014s
Aug 11, 2026 2:38:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 2:38:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 2:38:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 2:38:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 2:38:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 02:38:53.536067 3052 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/rangerkms_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/rangerkms_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.251.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.251.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 02:38:53 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415933, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 02:38:53.630896 3052 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 02:38:53.642189 3052 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 02:38:54.583909 3052 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 02:38:54.583995 3052 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms
I20260811 02:38:54.584013 3052 mini_ranger_kms.cc:227] Using postgres at 127.2.251.20:49433
2026-08-11 02:38:54,840 [I] DB FLAVOR :POSTGRES
2026-08-11 02:38:54,840 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 02:38:54,840 [I] Checking connection
2026-08-11 02:38:54,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 02:38:55,114 [I] connection success
2026-08-11 02:38:55,114 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 02:38:55,114 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 02:38:55,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 02:38:55,427 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 02:38:55,428 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 02:38:55,428 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 02:38:55,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:49433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 02:38:55,739 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 02:38:56.347426 3052 mini_ranger_kms.cc:310] Created kms service
I20260811 02:38:56.578970 3052 mini_ranger_kms.cc:326] Created kudu user
I20260811 02:38:56.634714 3052 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 02:38:56 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415933, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:38:57.123924 3052 mini_ranger_kms.cc:384] Added ranger policy
I20260811 02:38:57.124066 3052 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 02:38:57.124085 3052 mini_ranger_kms.cc:240] Using host: 127.2.251.20
I20260811 02:38:57.128476 3052 mini_ranger_kms.cc:276] Ranger KMS PID: 21913
I20260811 02:38:57.128628 3052 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.251.20:57783
Aug 11, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 2:38:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 02:38:59 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786415939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.251.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:38:59.721987 3052 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.091s user 0.006s sys 0.233s
I20260811 02:38:59.722117 3052 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 02:38:59.722215 3052 mini_ranger_kms.cc:401] 127.2.251.20:57783/kms/v1/keys
Aug 11 02:38:59 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.251.20@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
Aug 11 02:39:00 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.251.20@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
Aug 11 02:39:02 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:39:02.721835 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:33571
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:33571
--ranger_config_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:57783/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:02.826995 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:02.827229 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:02.827250 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:02.828441 22003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:39:02.828488 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:02.828501 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:02.828513 22003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:39:02.828537 22003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:39:02.830343 22003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.251.20:57783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:33571
--ranger_config_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:02.830621 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:02.830860 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:02.837069 22010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:02.837069 22009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:02.837049 22012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:02.837363 22003 server_base.cc:1034] running on GCE node
I20260811 02:39:02.837643 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:02.838004 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:02.839197 22003 hybrid_clock.cc:648] HybridClock initialized: now 1786415942839176 us; error 33 us; skew 500 ppm
Aug 11 02:39:02 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415942, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:02.847386 22003 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:39:02.847985 22003 webserver.cc:460] Webserver started at http://127.2.251.62:36219/ using document root <none> and password file <none>
I20260811 02:39:02.848244 22003 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:02.848286 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:02.848425 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:02 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415942, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:02.899647 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "40c55579e417451a8e85fb7ccd2cedb9"
format_stamp: "Formatted at 2026-08-11 02:39:02 on dist-test-slave-cvwc"
server_key: "8b7c93ebaab9f96f01c41c4f12d8ff95"
server_key_iv: "0d0e887db2668ad71a186652f750b558"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:02.900100 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "40c55579e417451a8e85fb7ccd2cedb9"
format_stamp: "Formatted at 2026-08-11 02:39:02 on dist-test-slave-cvwc"
server_key: "8b7c93ebaab9f96f01c41c4f12d8ff95"
server_key_iv: "0d0e887db2668ad71a186652f750b558"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:02.902069 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260811 02:39:02.915812 22022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:02.916038 22003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:39:02.916149 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "40c55579e417451a8e85fb7ccd2cedb9"
format_stamp: "Formatted at 2026-08-11 02:39:02 on dist-test-slave-cvwc"
server_key: "8b7c93ebaab9f96f01c41c4f12d8ff95"
server_key_iv: "0d0e887db2668ad71a186652f750b558"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:02.916260 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:02.927747 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:02.929673 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:02.929869 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:02.934628 22003 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:33571
I20260811 02:39:02.934648 22136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:33571 every 8 connection(s)
I20260811 02:39:02.935052 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:39:02.935994 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:02.938824 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9: Bootstrap starting.
I20260811 02:39:02.939458 22138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:02.939744 22138 log.cc:824] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:02.940478 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9: No bootstrap required, opened a new log
I20260811 02:39:02.941030 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 22003
I20260811 02:39:02.941136 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:39:02.941332 3052 external_mini_cluster.cc:1428] Setting key a156b9c18093d3452bee366538f2d5bf
I20260811 02:39:02.943164 22138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c55579e417451a8e85fb7ccd2cedb9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33571 } }
I20260811 02:39:02.943323 22138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:02.943358 22138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c55579e417451a8e85fb7ccd2cedb9, State: Initialized, Role: FOLLOWER
I20260811 02:39:02.943531 22138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c55579e417451a8e85fb7ccd2cedb9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33571 } }
I20260811 02:39:02.943611 22138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:39:02.943666 22138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:39:02.943722 22138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:02.944386 22138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c55579e417451a8e85fb7ccd2cedb9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33571 } }
I20260811 02:39:02.944494 22138 leader_election.cc:302] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c55579e417451a8e85fb7ccd2cedb9; no voters:
I20260811 02:39:02.944679 22138 leader_election.cc:288] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:39:02.944725 22155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:39:02.944921 22155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [term 1 LEADER]: Becoming Leader. State: Replica: 40c55579e417451a8e85fb7ccd2cedb9, State: Running, Role: LEADER
I20260811 02:39:02.945027 22138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:39:02.945052 22155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c55579e417451a8e85fb7ccd2cedb9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33571 } }
I20260811 02:39:02.945503 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c55579e417451a8e85fb7ccd2cedb9. Latest consensus state: current_term: 1 leader_uuid: "40c55579e417451a8e85fb7ccd2cedb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c55579e417451a8e85fb7ccd2cedb9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33571 } } }
I20260811 02:39:02.945583 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:39:02.945744 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c55579e417451a8e85fb7ccd2cedb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c55579e417451a8e85fb7ccd2cedb9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 33571 } } }
I20260811 02:39:02.945798 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:39:02.945901 22163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:39:02.946404 22163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:39:02.948745 22138 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 02:39:02.948923 22163 catalog_manager.cc:1347] Generated new cluster ID: ba41402d8278481a9a49712f57d9284d
I20260811 02:39:02.948971 22163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:39:02 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:02.967422 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:02.941921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.2.251.62:33571)
0811 02:39:02.942233 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:02.942243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:02.943157 (+ 914us) server_negotiation.cc:407] Connection header received
0811 02:39:02.943382 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:02.943390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:02.943452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:02.943565 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:02.944517 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:02.946155 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:02.947217 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:02.947809 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:02.947908 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:02.952467 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:02.952512 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:02.952512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:02.952597 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:02.955774 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:02.962356 (+ 6582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:02.962361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:02.962362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:02.962527 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:02.966133 (+ 3606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:02.966136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:02.966136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:02.966613 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:02.966804 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:02.966936 (+ 132us) server_negotiation.cc:299] Negotiation successful
0811 02:39:02.967131 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
I20260811 02:39:02.977828 22163 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:39:02.978776 22163 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:39:02.984697 22163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9: Generated new TSK 0
I20260811 02:39:02.984998 22163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:39:03.736433 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:57783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33571
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:03.836861 22228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:03.837102 22228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:03.837141 22228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:03.838678 22228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:03.838745 22228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:03.838788 22228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:39:03.840603 22228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.251.20:57783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:03.841248 22228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:03.841648 22228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:03.847311 22241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:03.847327 22247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:03.847646 22228 server_base.cc:1034] running on GCE node
W20260811 02:39:03.847251 22240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:03.847941 22228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:03.848232 22228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:03.849423 22228 hybrid_clock.cc:648] HybridClock initialized: now 1786415943849408 us; error 32 us; skew 500 ppm
Aug 11 02:39:03 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415943, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:03.852325 22228 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:39:03.852877 22228 webserver.cc:460] Webserver started at http://127.2.251.1:36083/ using document root <none> and password file <none>
I20260811 02:39:03.853125 22228 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:03.853175 22228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:03.853307 22228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:03 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415943, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:03.868552 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "fa1b9621079b43119ec7d43046ff7ceb"
format_stamp: "Formatted at 2026-08-11 02:39:03 on dist-test-slave-cvwc"
server_key: "9846ecc118593b043b5e08dc7677eaba"
server_key_iv: "4b642db71c2da73c34983fd7fc8b7231"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:03.869138 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "fa1b9621079b43119ec7d43046ff7ceb"
format_stamp: "Formatted at 2026-08-11 02:39:03 on dist-test-slave-cvwc"
server_key: "9846ecc118593b043b5e08dc7677eaba"
server_key_iv: "4b642db71c2da73c34983fd7fc8b7231"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:03.878291 22228 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.000s
I20260811 02:39:03.898625 22262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:03.898874 22228 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260811 02:39:03.898983 22228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "fa1b9621079b43119ec7d43046ff7ceb"
format_stamp: "Formatted at 2026-08-11 02:39:03 on dist-test-slave-cvwc"
server_key: "9846ecc118593b043b5e08dc7677eaba"
server_key_iv: "4b642db71c2da73c34983fd7fc8b7231"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:03.899111 22228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:03.921231 22228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:03.924041 22228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:03.924415 22228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:03.924813 22228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:03.925354 22228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:03.925467 22228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:03.925568 22228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:03.925671 22228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:03.933564 22228 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:42487
I20260811 02:39:03.937150 22228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:39:03.939316 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:42487 every 8 connection(s)
I20260811 02:39:03.944298 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 22228
I20260811 02:39:03.944419 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:39:03.944614 3052 external_mini_cluster.cc:1428] Setting key b26cc6eb3273112e117422f65c5dc090
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Aug 11 02:39:03 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415943, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:03.962450 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:03.938245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:57581 (local address 127.2.251.62:33571)
0811 02:39:03.938412 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:03.938414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:03.950181 (+ 11767us) server_negotiation.cc:407] Connection header received
0811 02:39:03.951813 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:03.951815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:03.951840 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:03.951929 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:03.952868 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:03.953637 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:03.955251 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:03.955567 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:03.955618 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:03.958478 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:03.958480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:03.958481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:03.958490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:03.960272 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:03.961073 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:03.961076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:03.961076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:03.961143 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:03.961428 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:03.961429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:03.961429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:03.961594 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:03.961646 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:03.962165 (+ 519us) server_negotiation.cc:299] Negotiation successful
0811 02:39:03.962269 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
I20260811 02:39:03.965811 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:03.938228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33571 (local address 127.2.251.1:57581)
0811 02:39:03.949966 (+ 11738us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:03.950047 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:03.950739 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:03.952099 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:03.952104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:03.952421 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:03.952736 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:03.952743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:03.953802 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:03.953803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:03.955136 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:03.955140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:03.955721 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:03.955722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:03.955876 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:03.956505 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:03.956517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:03.958329 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:03.960630 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:03.960641 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:03.960644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:03.960972 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:03.961205 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:03.961206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:03.961206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:03.961347 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:03.961680 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:03.961687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:03.961986 (+ 299us) client_negotiation.cc:769] Sending connection context
0811 02:39:03.965247 (+ 3261us) client_negotiation.cc:240] Negotiation successful
0811 02:39:03.965495 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11659,"thread_start_us":107,"threads_started":1}
I20260811 02:39:03.966634 22458 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33571
I20260811 02:39:03.966734 22458 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:03.966940 22458 heartbeater.cc:511] Master 127.2.251.62:33571 requested a full tablet report, sending...
I20260811 02:39:03.967583 22060 ts_manager.cc:194] Registered new tserver with Master: fa1b9621079b43119ec7d43046ff7ceb (127.2.251.1:42487)
I20260811 02:39:03.968467 22060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:57581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:39:03.986553 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:57783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33571
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:04.117081 22472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:04.117479 22472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:04.117566 22472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:04.119583 22472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:04.119848 22472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:04.119957 22472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:39:04.122491 22472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.251.20:57783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:04.122996 22472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:04.123327 22472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:04.130877 22485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:04.132169 22472 server_base.cc:1034] running on GCE node
W20260811 02:39:04.131927 22492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:04.132982 22486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:04.133348 22472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:04.133750 22472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:04.134953 22472 hybrid_clock.cc:648] HybridClock initialized: now 1786415944134926 us; error 44 us; skew 500 ppm
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:04.138867 22472 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:39:04.140359 22472 webserver.cc:460] Webserver started at http://127.2.251.2:34347/ using document root <none> and password file <none>
I20260811 02:39:04.140811 22472 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:04.140970 22472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:04.141206 22472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:04.165416 22472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "072dfdeb32064410b7dbc19415039c98"
format_stamp: "Formatted at 2026-08-11 02:39:04 on dist-test-slave-cvwc"
server_key: "cef96b37972d0b1718a36e26c7ecd62c"
server_key_iv: "cba5d5fca0d8e6edf2cae1b014fd1885"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:04.166098 22472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "072dfdeb32064410b7dbc19415039c98"
format_stamp: "Formatted at 2026-08-11 02:39:04 on dist-test-slave-cvwc"
server_key: "cef96b37972d0b1718a36e26c7ecd62c"
server_key_iv: "cba5d5fca0d8e6edf2cae1b014fd1885"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:04.171164 22472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:04.196981 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:04.197249 22472 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260811 02:39:04.197379 22472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "072dfdeb32064410b7dbc19415039c98"
format_stamp: "Formatted at 2026-08-11 02:39:04 on dist-test-slave-cvwc"
server_key: "cef96b37972d0b1718a36e26c7ecd62c"
server_key_iv: "cba5d5fca0d8e6edf2cae1b014fd1885"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:04.197495 22472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:04.243083 22472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:04.255892 22472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:04.256145 22472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:04.256402 22472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:04.257130 22472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:04.257261 22472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 02:39:04.257336 22472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:04.257392 22472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:04.264214 22472 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:44401
I20260811 02:39:04.264488 22698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:44401 every 8 connection(s)
I20260811 02:39:04.265575 22472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:39:04.272431 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 22472
I20260811 02:39:04.272560 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:39:04.272768 3052 external_mini_cluster.cc:1428] Setting key e4d3411dbd07213d3289440cedc6fc06
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:04.282868 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.270589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33571 (local address 127.2.251.2:53605)
0811 02:39:04.270871 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:04.270929 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:04.271632 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:04.271974 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:04.271980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:04.272299 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:04.272621 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.272629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.273698 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.273699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.275107 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.275112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.275708 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.275709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.275866 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.276586 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:04.276597 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:04.278811 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:04.280877 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:04.280889 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:04.280893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:04.281260 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:04.281530 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:04.281531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:04.281531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:04.281686 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:04.282083 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:04.282090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:04.282304 (+ 214us) client_negotiation.cc:769] Sending connection context
0811 02:39:04.282411 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 02:39:04.282594 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":129,"threads_started":1}
I20260811 02:39:04.283814 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.270962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:53605 (local address 127.2.251.62:33571)
0811 02:39:04.271037 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:04.271039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:04.271050 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:39:04.271769 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:04.271771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:04.271786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:04.271859 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:04.272761 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.273557 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.275224 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.275541 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.275591 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.278968 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:04.278971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:04.278972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:04.278982 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:04.280778 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:04.281367 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:04.281369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:04.281369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:04.281439 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:04.281817 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:04.281818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:04.281818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:04.281979 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:04.282042 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:04.283594 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0811 02:39:04.283699 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 02:39:04.284288 22699 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33571
I20260811 02:39:04.284410 22699 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:04.284650 22699 heartbeater.cc:511] Master 127.2.251.62:33571 requested a full tablet report, sending...
I20260811 02:39:04.285120 22060 ts_manager.cc:194] Registered new tserver with Master: 072dfdeb32064410b7dbc19415039c98 (127.2.251.2:44401)
I20260811 02:39:04.285629 22060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:53605
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:39:04.314476 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:57783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33571
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:04.453482 22713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:04.454031 22713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:04.454137 22713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:04.456252 22713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:04.456589 22713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:04.456718 22713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:39:04.459470 22713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.251.20:57783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:04.460026 22713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:04.460424 22713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:04.467907 22722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:04.468367 22721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:04.468641 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:04.469656 22713 server_base.cc:1034] running on GCE node
I20260811 02:39:04.470041 22713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:04.470358 22713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:04.501976 22713 hybrid_clock.cc:648] HybridClock initialized: now 1786415944501914 us; error 109 us; skew 500 ppm
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:04.506258 22713 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:39:04.506959 22713 webserver.cc:460] Webserver started at http://127.2.251.3:33845/ using document root <none> and password file <none>
I20260811 02:39:04.507314 22713 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:04.507449 22713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:04.507658 22713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:04.523356 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "26cc895e93fc43119419b41b36badc4c"
format_stamp: "Formatted at 2026-08-11 02:39:04 on dist-test-slave-cvwc"
server_key: "0d26fdf0413e51c0de18c1d8ef5567d4"
server_key_iv: "64ad40345b13e973f70da687106172c5"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:04.523902 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "26cc895e93fc43119419b41b36badc4c"
format_stamp: "Formatted at 2026-08-11 02:39:04 on dist-test-slave-cvwc"
server_key: "0d26fdf0413e51c0de18c1d8ef5567d4"
server_key_iv: "64ad40345b13e973f70da687106172c5"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:04.526747 22713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 02:39:04.541169 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:04.541438 22713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:39:04.541546 22713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "26cc895e93fc43119419b41b36badc4c"
format_stamp: "Formatted at 2026-08-11 02:39:04 on dist-test-slave-cvwc"
server_key: "0d26fdf0413e51c0de18c1d8ef5567d4"
server_key_iv: "64ad40345b13e973f70da687106172c5"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:04.541644 22713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:04.565521 22713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:04.568214 22713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:04.568452 22713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:04.568771 22713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:04.569235 22713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:04.569327 22713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:04.569404 22713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:04.569466 22713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:04.580714 22713 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:38249
I20260811 02:39:04.581418 22713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:39:04.582307 22935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:38249 every 8 connection(s)
I20260811 02:39:04.583773 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 22713
I20260811 02:39:04.583896 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:39:04.584098 3052 external_mini_cluster.cc:1428] Setting key 270cd7da6b147beaf432ebf2c57f4dfe
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:04.596170 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.583144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:45837 (local address 127.2.251.62:33571)
0811 02:39:04.583198 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:04.583200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:04.583495 (+ 295us) server_negotiation.cc:407] Connection header received
0811 02:39:04.585067 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:04.585069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:04.585082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:04.585114 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:04.586084 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.586973 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.588638 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.588956 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.589005 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.591918 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:04.591920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:04.591920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:04.591929 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:04.593668 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:04.594306 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:04.594308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:04.594308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:04.594378 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:04.594694 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:04.594696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:04.594696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:04.594862 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:04.594921 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:04.595962 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0811 02:39:04.596064 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:39:04.597050 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.583152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:33571 (local address 127.2.251.3:45837)
0811 02:39:04.583342 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:04.583391 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:04.584306 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:04.585338 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:04.585343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:04.585616 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:04.585966 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.585973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.587252 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.587253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.588525 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.588529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.589111 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.589112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.589267 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.589970 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:04.589982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:04.591788 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:04.593842 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:04.593853 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:04.593857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:04.594207 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:04.594454 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:04.594455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:04.594455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:04.594609 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:04.594961 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:04.594969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:04.595178 (+ 209us) client_negotiation.cc:769] Sending connection context
0811 02:39:04.596567 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0811 02:39:04.596768 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260811 02:39:04.597767 22937 heartbeater.cc:347] Connected to a master server at 127.2.251.62:33571
I20260811 02:39:04.597875 22937 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:04.598088 22937 heartbeater.cc:511] Master 127.2.251.62:33571 requested a full tablet report, sending...
I20260811 02:39:04.598567 22060 ts_manager.cc:194] Registered new tserver with Master: 26cc895e93fc43119419b41b36badc4c (127.2.251.3:38249)
I20260811 02:39:04.599094 22060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:45837
I20260811 02:39:04.606815 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:39:04.628808 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.614167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.2.251.62:33571)
0811 02:39:04.614231 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:04.614233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:04.614464 (+ 231us) server_negotiation.cc:407] Connection header received
0811 02:39:04.614578 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:04.614580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:04.614612 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:04.614643 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:04.617897 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.618726 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.621032 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.621363 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.621421 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.624903 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:04.624905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:04.624906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:04.624917 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:04.626774 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:04.627740 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:04.627742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:04.627742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:04.627831 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:04.628072 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:04.628073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:04.628073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:04.628235 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:04.628323 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:04.628580 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 02:39:04.628683 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 02:39:04.631587 22060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:39:04.634650 22060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:39:04.664862 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.642022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42487 (local address 127.0.0.1:46794)
0811 02:39:04.647296 (+ 5274us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:04.647307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:04.647348 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:04.648545 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:04.648549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:04.648552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:04.648652 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.648654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.659710 (+ 11056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.659712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.660763 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.660768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.664482 (+ 3714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.664484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.664594 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.664611 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:39:04.664683 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 02:39:04.664767 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5244,"thread_start_us":100,"threads_started":1}
I20260811 02:39:04.665702 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.643697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.2.251.3:38249)
0811 02:39:04.647975 (+ 4278us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:04.647980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:04.647989 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:39:04.648026 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:04.648028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:04.648043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:04.648113 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:04.648857 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.652520 (+ 3663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.662450 (+ 9930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.663275 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.663349 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.663432 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:04.665562 (+ 2130us) server_negotiation.cc:299] Negotiation successful
0811 02:39:04.665621 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4239,"thread_start_us":159,"threads_started":1}
I20260811 02:39:04.665478 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.643159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:38249 (local address 127.0.0.1:53008)
0811 02:39:04.647799 (+ 4640us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:04.647809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:04.647852 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:04.648736 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:04.648737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:04.648739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:04.648770 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.648771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.652642 (+ 3871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.652643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.662309 (+ 9666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.662314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.665221 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.665222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.665292 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.665298 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:39:04.665352 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 02:39:04.665401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4612,"thread_start_us":102,"threads_started":1}
I20260811 02:39:04.667613 22823 tablet_service.cc:1467] Processing CreateTablet for tablet 1658596dca04481b9fb54188b4181a45 (DEFAULT_TABLE table=test-table [id=e29ad7972a8545febe48b226371c61a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:39:04.668089 22823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1658596dca04481b9fb54188b4181a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:04.666298 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.641525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:44401 (local address 127.0.0.1:59322)
0811 02:39:04.646549 (+ 5024us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:04.646627 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:04.647159 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:04.650095 (+ 2936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:04.650096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:04.650098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:04.650148 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.650150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.654751 (+ 4601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.654752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.658953 (+ 4201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:04.658958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.666046 (+ 7088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.666047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:04.666112 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.666116 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:39:04.666165 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 02:39:04.666215 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4973,"thread_start_us":239,"threads_started":1}
I20260811 02:39:04.673233 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.642469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.2.251.2:44401)
0811 02:39:04.648289 (+ 5820us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:04.648294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:04.648305 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:39:04.648344 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:04.648346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:04.648362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:04.648434 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:04.650243 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.651376 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.660918 (+ 9542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.661852 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.661930 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.662018 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:04.673054 (+ 11036us) server_negotiation.cc:299] Negotiation successful
0811 02:39:04.673129 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5769,"thread_start_us":121,"threads_started":1}
I20260811 02:39:04.673635 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:04.642712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.2.251.1:42487)
0811 02:39:04.647521 (+ 4809us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:04.647528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:04.647540 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:39:04.647579 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:04.647581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:04.647599 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:04.647672 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:04.649007 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.659592 (+ 10585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.663484 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:04.664307 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:04.664380 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:04.664436 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:04.673487 (+ 9051us) server_negotiation.cc:299] Negotiation successful
0811 02:39:04.673558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4761,"thread_start_us":87,"threads_started":1}
I20260811 02:39:04.674624 22587 tablet_service.cc:1467] Processing CreateTablet for tablet 1658596dca04481b9fb54188b4181a45 (DEFAULT_TABLE table=test-table [id=e29ad7972a8545febe48b226371c61a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:39:04.674738 22342 tablet_service.cc:1467] Processing CreateTablet for tablet 1658596dca04481b9fb54188b4181a45 (DEFAULT_TABLE table=test-table [id=e29ad7972a8545febe48b226371c61a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:39:04.675041 22587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1658596dca04481b9fb54188b4181a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:04.675107 22342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1658596dca04481b9fb54188b4181a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:04.681375 22979 tablet_bootstrap.cc:492] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: Bootstrap starting.
I20260811 02:39:04.681676 22978 tablet_bootstrap.cc:492] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Bootstrap starting.
I20260811 02:39:04.681998 22975 tablet_bootstrap.cc:492] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Bootstrap starting.
I20260811 02:39:04.683162 22979 tablet_bootstrap.cc:654] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:04.684463 22975 tablet_bootstrap.cc:654] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:04.685643 22978 tablet_bootstrap.cc:654] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:04.686177 22979 log.cc:824] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:04.686717 22978 log.cc:824] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:04.687225 22975 log.cc:824] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:04.689638 22979 tablet_bootstrap.cc:492] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: No bootstrap required, opened a new log
I20260811 02:39:04.689836 22979 ts_tablet_manager.cc:1397] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260811 02:39:04.692971 22979 raft_consensus.cc:348] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:04.693393 22979 raft_consensus.cc:374] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:04.693498 22979 raft_consensus.cc:729] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1b9621079b43119ec7d43046ff7ceb, State: Initialized, Role: FOLLOWER
I20260811 02:39:04.693786 22979 consensus_queue.cc:260] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:04.694298 22979 ts_tablet_manager.cc:1428] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 02:39:04.694533 22458 heartbeater.cc:503] Master 127.2.251.62:33571 was elected leader, sending a full tablet report...
I20260811 02:39:04.696321 22978 tablet_bootstrap.cc:492] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: No bootstrap required, opened a new log
I20260811 02:39:04.696480 22978 ts_tablet_manager.cc:1397] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260811 02:39:04.698876 22978 raft_consensus.cc:348] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:04.699180 22978 raft_consensus.cc:374] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:04.699298 22978 raft_consensus.cc:729] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072dfdeb32064410b7dbc19415039c98, State: Initialized, Role: FOLLOWER
I20260811 02:39:04.699539 22978 consensus_queue.cc:260] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:04.699918 22978 ts_tablet_manager.cc:1428] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:39:04.700145 22699 heartbeater.cc:503] Master 127.2.251.62:33571 was elected leader, sending a full tablet report...
I20260811 02:39:04.702113 22975 tablet_bootstrap.cc:492] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: No bootstrap required, opened a new log
I20260811 02:39:04.702215 22975 ts_tablet_manager.cc:1397] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20260811 02:39:04.704938 22975 raft_consensus.cc:348] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:04.705220 22975 raft_consensus.cc:374] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:04.705264 22975 raft_consensus.cc:729] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cc895e93fc43119419b41b36badc4c, State: Initialized, Role: FOLLOWER
I20260811 02:39:04.705797 22975 consensus_queue.cc:260] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:04.706274 22975 ts_tablet_manager.cc:1428] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 02:39:04.706468 22937 heartbeater.cc:503] Master 127.2.251.62:33571 was elected leader, sending a full tablet report...
Aug 11 02:39:04 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
W20260811 02:39:04.766959 22703 tablet.cc:2369] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:39:04.832487 22941 tablet.cc:2369] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:39:04.938612 22460 tablet.cc:2369] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:39:05.030532 22989 raft_consensus.cc:482] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:39:05.030689 22989 raft_consensus.cc:504] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:05.031152 22989 leader_election.cc:288] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1b9621079b43119ec7d43046ff7ceb (127.2.251.1:42487), 26cc895e93fc43119419b41b36badc4c (127.2.251.3:38249)
Aug 11 02:39:05 dist-test-slave-cvwc krb5kdc[21195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:05.036588 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.031862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:59243 (local address 127.2.251.1:42487)
0811 02:39:05.031905 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:05.031907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:05.032090 (+ 183us) server_negotiation.cc:407] Connection header received
0811 02:39:05.032993 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:05.032995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:05.033006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:05.033036 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:05.033250 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.034010 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.035650 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.036171 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.036217 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:05.036231 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:05.036453 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 02:39:05.036505 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:39:05.037297 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.031395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42487 (local address 127.2.251.2:59243)
0811 02:39:05.032018 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:05.032030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:05.032121 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:05.033104 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:05.033105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:05.033107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:05.033176 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:05.033179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.034117 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.034118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:05.035516 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:05.035519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.036292 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.036293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:05.036368 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:05.036372 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:39:05.037150 (+ 778us) client_negotiation.cc:240] Negotiation successful
0811 02:39:05.037207 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":99,"threads_started":1}
I20260811 02:39:05.037850 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658596dca04481b9fb54188b4181a45" candidate_uuid: "072dfdeb32064410b7dbc19415039c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1b9621079b43119ec7d43046ff7ceb" is_pre_election: true
I20260811 02:39:05.038049 22358 raft_consensus.cc:2454] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072dfdeb32064410b7dbc19415039c98 in term 0.
I20260811 02:39:05.038354 22506 leader_election.cc:302] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072dfdeb32064410b7dbc19415039c98, fa1b9621079b43119ec7d43046ff7ceb; no voters:
I20260811 02:39:05.039265 22989 raft_consensus.cc:2781] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:39:05.039330 22989 raft_consensus.cc:482] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:39:05.039351 22989 raft_consensus.cc:3037] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:05.040239 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.031718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:50165 (local address 127.2.251.3:38249)
0811 02:39:05.031766 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:05.031768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:05.032297 (+ 529us) server_negotiation.cc:407] Connection header received
0811 02:39:05.032402 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:05.032404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:05.032415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:05.032444 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:05.032679 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.033407 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.039385 (+ 5978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.039893 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.039980 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:05.039993 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:05.040128 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 02:39:05.040173 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:39:05.040248 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.031630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:38249 (local address 127.2.251.2:50165)
0811 02:39:05.032226 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:05.032235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:05.032326 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:05.032512 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:05.032513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:05.032515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:05.032583 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:05.032585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.037505 (+ 4920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.037506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:05.039257 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:05.039261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.039993 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.039994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:05.040054 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:05.040059 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:39:05.040111 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 02:39:05.040152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":114,"threads_started":1}
I20260811 02:39:05.040318 22989 raft_consensus.cc:504] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:05.040479 22989 leader_election.cc:288] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [CANDIDATE]: Term 1 election: Requested vote from peers fa1b9621079b43119ec7d43046ff7ceb (127.2.251.1:42487), 26cc895e93fc43119419b41b36badc4c (127.2.251.3:38249)
I20260811 02:39:05.040617 22358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658596dca04481b9fb54188b4181a45" candidate_uuid: "072dfdeb32064410b7dbc19415039c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1b9621079b43119ec7d43046ff7ceb"
I20260811 02:39:05.040657 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658596dca04481b9fb54188b4181a45" candidate_uuid: "072dfdeb32064410b7dbc19415039c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cc895e93fc43119419b41b36badc4c" is_pre_election: true
I20260811 02:39:05.040704 22358 raft_consensus.cc:3037] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:05.040793 22854 raft_consensus.cc:2454] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 072dfdeb32064410b7dbc19415039c98 in term 0.
I20260811 02:39:05.040918 22852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658596dca04481b9fb54188b4181a45" candidate_uuid: "072dfdeb32064410b7dbc19415039c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cc895e93fc43119419b41b36badc4c"
I20260811 02:39:05.040982 22852 raft_consensus.cc:3037] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:05.041628 22358 raft_consensus.cc:2454] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072dfdeb32064410b7dbc19415039c98 in term 1.
I20260811 02:39:05.041672 22852 raft_consensus.cc:2454] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 072dfdeb32064410b7dbc19415039c98 in term 1.
I20260811 02:39:05.041875 22506 leader_election.cc:302] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072dfdeb32064410b7dbc19415039c98, fa1b9621079b43119ec7d43046ff7ceb; no voters:
I20260811 02:39:05.049834 22989 raft_consensus.cc:2781] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:39:05.050107 22989 raft_consensus.cc:686] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 1 LEADER]: Becoming Leader. State: Replica: 072dfdeb32064410b7dbc19415039c98, State: Running, Role: LEADER
I20260811 02:39:05.050244 22989 consensus_queue.cc:237] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } }
I20260811 02:39:05.051327 22060 catalog_manager.cc:5495] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 072dfdeb32064410b7dbc19415039c98 (127.2.251.2). New cstate: current_term: 1 leader_uuid: "072dfdeb32064410b7dbc19415039c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072dfdeb32064410b7dbc19415039c98" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 } health_report { overall_health: UNKNOWN } } }
I20260811 02:39:05.088362 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.085008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.2.251.2:44401)
0811 02:39:05.085055 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:05.085057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:05.085064 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:39:05.085107 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:05.085108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:05.085132 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:05.085201 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:39:05.085382 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.086182 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.087280 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:05.087673 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:05.087769 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:05.087957 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:39:05.088107 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:39:05.088200 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:05.088224 (+ 24us) server_negotiation.cc:299] Negotiation successful
0811 02:39:05.088250 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:39:05.090528 22852 raft_consensus.cc:1261] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 1 FOLLOWER]: Refusing update from remote peer 072dfdeb32064410b7dbc19415039c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:39:05.090584 22358 raft_consensus.cc:1261] T 1658596dca04481b9fb54188b4181a45 P fa1b9621079b43119ec7d43046ff7ceb [term 1 FOLLOWER]: Refusing update from remote peer 072dfdeb32064410b7dbc19415039c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:39:05.090914 23001 consensus_queue.cc:1035] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cc895e93fc43119419b41b36badc4c" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 38249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:39:05.091068 22989 consensus_queue.cc:1035] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1b9621079b43119ec7d43046ff7ceb" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:39:05.094254 23007 mvcc.cc:205] Tried to move back new op lower bound from 7317159711087677440 to 7317159710926483456. Current Snapshot: MvccSnapshot[applied={T|T < 7317159711087677440}]
I20260811 02:39:05.095481 23011 mvcc.cc:205] Tried to move back new op lower bound from 7317159711087677440 to 7317159710926483456. Current Snapshot: MvccSnapshot[applied={T|T < 7317159711087677440}]
I20260811 02:39:05.099498 23008 mvcc.cc:205] Tried to move back new op lower bound from 7317159711087677440 to 7317159710926483456. Current Snapshot: MvccSnapshot[applied={T|T < 7317159711087677440}]
I20260811 02:39:05.101150 22060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:39:05.101267 22060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:39:05.102154 22060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40c55579e417451a8e85fb7ccd2cedb9: Sending DeleteTablet for 3 replicas of tablet 1658596dca04481b9fb54188b4181a45
I20260811 02:39:05.102572 22587 tablet_service.cc:1514] Processing DeleteTablet for tablet 1658596dca04481b9fb54188b4181a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:39:05 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59322
I20260811 02:39:05.102646 22342 tablet_service.cc:1514] Processing DeleteTablet for tablet 1658596dca04481b9fb54188b4181a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:39:05 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:46794
I20260811 02:39:05.103036 22823 tablet_service.cc:1514] Processing DeleteTablet for tablet 1658596dca04481b9fb54188b4181a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:39:05 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:53008
I20260811 02:39:05.103245 23028 tablet_replica.cc:330] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: stopping tablet replica
I20260811 02:39:05.103600 23028 raft_consensus.cc:2229] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:39:05.103730 23028 raft_consensus.cc:2258] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:39:05.103783 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 22228
W20260811 02:39:05.142372 22023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:42487 (error 108)
I20260811 02:39:05.145910 23021 tablet_replica.cc:330] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: stopping tablet replica
I20260811 02:39:05.146457 23021 raft_consensus.cc:2229] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:39:05.147593 23021 raft_consensus.cc:2258] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:39:05.148294 23021 ts_tablet_manager.cc:1905] T 1658596dca04481b9fb54188b4181a45 P 072dfdeb32064410b7dbc19415039c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:39:05.153877 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 22472
I20260811 02:39:05.185279 23028 ts_tablet_manager.cc:1905] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:39:05.187579 23028 ts_tablet_manager.cc:1918] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:39:05.187757 23028 log.cc:1196] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/1658596dca04481b9fb54188b4181a45
I20260811 02:39:05.188159 23028 ts_tablet_manager.cc:1939] T 1658596dca04481b9fb54188b4181a45 P 26cc895e93fc43119419b41b36badc4c: Deleting consensus metadata
I20260811 02:39:05.188808 22024 catalog_manager.cc:4841] TS 26cc895e93fc43119419b41b36badc4c (127.2.251.3:38249): tablet 1658596dca04481b9fb54188b4181a45 (table test-table [id=e29ad7972a8545febe48b226371c61a9]) successfully deleted
I20260811 02:39:05.188999 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.182922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42487 (local address 127.0.0.1:46796)
0811 02:39:05.188843 (+ 5921us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:05.188925 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5887,"thread_start_us":867,"threads_started":1}
W20260811 02:39:05.189154 22023 catalog_manager.cc:4584] TS fa1b9621079b43119ec7d43046ff7ceb (127.2.251.1:42487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1658596dca04481b9fb54188b4181a45: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42487: connect: Connection refused (error 111)
W20260811 02:39:05.219790 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:39:05.219434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:44401 (local address 127.0.0.1:59344)
0811 02:39:05.219502 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:05.219513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:05.219567 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:05.219701 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.2:44401: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 02:39:05.220559 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 22713
I20260811 02:39:05.252413 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.252273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42487 (local address 127.0.0.1:46802)
0811 02:39:05.252331 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:05.252362 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 02:39:05.252974 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:05.252850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42487 (local address 127.0.0.1:46816)
0811 02:39:05.252912 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:05.252937 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 02:39:05.258843 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 22003
I20260811 02:39:05.284293 3052 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 02:39:05.631259 3052 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 02:39:05.631384 3052 mini_ranger.cc:66] Stopping Ranger...
I20260811 02:39:05.987701 3052 mini_ranger.cc:68] Stopped Ranger
2026-08-11 02:39:05.987 UTC [21253] LOG: received smart shutdown request
2026-08-11 02:39:05.992 UTC [21253] LOG: background worker "logical replication launcher" (PID 21268) exited with exit code 1
2026-08-11 02:39:05.993 UTC [21262] LOG: shutting down
2026-08-11 02:39:06.222 UTC [21253] LOG: database system is shut down
2026-08-11T02:39:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51734 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:39:06 dist-test-slave-cvwc krb5kdc[23050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:39:06 dist-test-slave-cvwc krb5kdc[23050](info): set up 2 sockets
Aug 11 02:39:06 dist-test-slave-cvwc krb5kdc[23050](info): commencing operation
krb5kdc: starting...
W20260811 02:39:07.550186 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:39:07 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:39:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:39:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/rangeradmin_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/rangeradmin_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/rangerlookup_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/rangerlookup_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
I20260811 02:39:07.752255 3052 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRZ17kF/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/postgres -l logfile start
2026-08-11 02:39:10.720 UTC [23100] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 02:39:10.720 UTC [23100] LOG: listening on IPv4 address "127.2.251.20", port 36187
2026-08-11 02:39:10.724 UTC [23100] LOG: listening on Unix socket "/tmp/.s.PGSQL.36187"
2026-08-11 02:39:10.747 UTC [23106] LOG: database system was shut down at 2026-08-11 02:39:08 UTC
2026-08-11 02:39:10.752 UTC [23100] LOG: database system is ready to accept connections
I20260811 02:39:11.871417 3052 mini_postgres.cc:96] Postgres bound to 36187
2026-08-11 02:39:11.880 UTC [23121] FATAL: database "slave" does not exist
127.2.251.20:36187 - accepting connections
I20260811 02:39:11.880832 3052 mini_ranger.cc:161] Starting Ranger...
I20260811 02:39:11.897168 3052 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 02:39:12.465744 3052 mini_ranger.cc:87] Created ranger Postgres database
I20260811 02:39:12.465837 3052 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-admin
2026-08-11 02:39:12,781 [I] DB FLAVOR :POSTGRES
2026-08-11 02:39:12,782 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 02:39:12,782 [I] Checking connection..
2026-08-11 02:39:12,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 02:39:13,048 [I] Checking connection passed.
2026-08-11 02:39:13,048 [I] --------- Verifying version history table ---------
2026-08-11 02:39:13,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 02:39:13,336 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 02:39:13,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 02:39:13,617 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 02:39:13,618 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 02:39:13,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 02:39:13.881 UTC [23256] WARNING: there is no transaction in progress
2026-08-11 02:39:13,889 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 02:39:13,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 02:39:14,180 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 02:39:14,180 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 02:39:14,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 02:39:14,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 02:39:14,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cvwc.c.gcp-upstream.internal','N') ;"
2026-08-11 02:39:15,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 02:39:15,293 [I] Table x_portal_user does not exist in database ranger
2026-08-11 02:39:15,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 02:39:15,561 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 02:39:15,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 02:39:15,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 02:39:16,088 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 02:39:16,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 02:39:17.305 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.326 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.346 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.366 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.384 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.411 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.502 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.512 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.534 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.548 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.560 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.577 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.587 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.597 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.606 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:17.616 UTC [23447] WARNING: there is no transaction in progress
2026-08-11 02:39:18,419 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 02:39:18,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 02:39:18,738 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 02:39:18,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 02:39:19,024 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 02:39:19,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 02:39:19,324 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 02:39:19,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 02:39:19,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 02:39:19,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cvwc.c.gcp-upstream.internal';"
2026-08-11 02:39:20,124 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 02:39:20,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 02:39:20,408 [I] DB_PATCHES have already been applied
I20260811 02:39:20.414922 3052 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRZ17kF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 02:39:20.415004 3052 mini_ranger.cc:193] Using host: 127.2.251.20
I20260811 02:39:20.419242 3052 mini_ranger.cc:227] Ranger admin URL: http://127.2.251.20:54451
Aug 11, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 2:39:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 2:39:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 2:39:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 02:39:21.781221 3052 mini_ranger.cc:160] Time spent starting Ranger: real 9.900s user 0.000s sys 0.011s
Aug 11, 2026 2:39:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 2:39:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 2:39:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 2:39:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 2:39:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 02:39:45.073154 3052 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.251.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.251.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/rangerkms_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.251.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/rangerkms_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.251.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.251.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.251.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/HTTP_127.2.251.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 02:39:45 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415985, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 02:39:45.162689 3052 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 02:39:45.173483 3052 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 02:39:46.083020 3052 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 02:39:46.083114 3052 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms
I20260811 02:39:46.083132 3052 mini_ranger_kms.cc:227] Using postgres at 127.2.251.20:36187
2026-08-11 02:39:46,262 [I] DB FLAVOR :POSTGRES
2026-08-11 02:39:46,262 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 02:39:46,263 [I] Checking connection
2026-08-11 02:39:46,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 02:39:46,545 [I] connection success
2026-08-11 02:39:46,545 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 02:39:46,545 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 02:39:46,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 02:39:46,820 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 02:39:46,820 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 02:39:46,821 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 02:39:46,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.251.20:36187/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 02:39:47,170 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 02:39:47.693243 3052 mini_ranger_kms.cc:310] Created kms service
I20260811 02:39:47.895454 3052 mini_ranger_kms.cc:326] Created kudu user
I20260811 02:39:47.942881 3052 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 02:39:47 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415985, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:48.377177 3052 mini_ranger_kms.cc:384] Added ranger policy
I20260811 02:39:48.377342 3052 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRZ17kF/build/release/bin/postgresql.jar:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRZ17kF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 02:39:48.377373 3052 mini_ranger_kms.cc:240] Using host: 127.2.251.20
I20260811 02:39:48.384799 3052 mini_ranger_kms.cc:276] Ranger KMS PID: 23743
I20260811 02:39:48.384989 3052 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.251.20:35731
Aug 11, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 2:39:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 2:39:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 2:39:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 02:39:50 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786415990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.251.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:50.923815 3052 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.761s user 0.009s sys 0.154s
I20260811 02:39:50.923914 3052 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 02:39:50.923964 3052 mini_ranger_kms.cc:401] 127.2.251.20:35731/kms/v1/keys
Aug 11 02:39:51 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.251.20@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
Aug 11 02:39:51 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.251.20@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
Aug 11 02:39:54 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:39:54.128449 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42217
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:42217
--ranger_config_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:54.233429 23831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:54.233659 23831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:54.233690 23831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:54.234882 23831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:39:54.234939 23831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:54.234959 23831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:54.234975 23831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:39:54.234992 23831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:39:54.236639 23831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.251.20:35731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:42217
--ranger_config_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:54.236891 23831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:54.237109 23831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:54.242550 23839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:54.242600 23838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:54.242619 23845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:54.242628 23831 server_base.cc:1034] running on GCE node
I20260811 02:39:54.243022 23831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:54.243326 23831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:54.244493 23831 hybrid_clock.cc:648] HybridClock initialized: now 1786415994244463 us; error 48 us; skew 500 ppm
Aug 11 02:39:54 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415994, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:54.247213 23831 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:39:54.247704 23831 webserver.cc:460] Webserver started at http://127.2.251.62:36549/ using document root <none> and password file <none>
I20260811 02:39:54.247920 23831 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:54.247963 23831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:54.248091 23831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:54 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415994, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:54.295694 23831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "40b1cef8eac445c58972f4d02640a403"
format_stamp: "Formatted at 2026-08-11 02:39:54 on dist-test-slave-cvwc"
server_key: "26200fe0c11fafa013c020ff77c76e32"
server_key_iv: "c636b33470539c1076d88993020278ea"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:54.296295 23831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "40b1cef8eac445c58972f4d02640a403"
format_stamp: "Formatted at 2026-08-11 02:39:54 on dist-test-slave-cvwc"
server_key: "26200fe0c11fafa013c020ff77c76e32"
server_key_iv: "c636b33470539c1076d88993020278ea"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:54.298504 23831 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 02:39:54.316378 23864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:54.316659 23831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:39:54.316778 23831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "40b1cef8eac445c58972f4d02640a403"
format_stamp: "Formatted at 2026-08-11 02:39:54 on dist-test-slave-cvwc"
server_key: "26200fe0c11fafa013c020ff77c76e32"
server_key_iv: "c636b33470539c1076d88993020278ea"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:54.316890 23831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:54.353921 23831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:54.356292 23831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:54.356459 23831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:54.361747 23831 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:42217
I20260811 02:39:54.361826 23952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:42217 every 8 connection(s)
I20260811 02:39:54.362236 23831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:39:54.363013 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:54.365691 23953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Bootstrap starting.
I20260811 02:39:54.366477 23953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:54.366789 23953 log.cc:824] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:54.367527 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 23831
I20260811 02:39:54.367645 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:39:54.367904 3052 external_mini_cluster.cc:1428] Setting key 0c0a25caeb35858a39ea0ad55ded4418
I20260811 02:39:54.367976 23953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: No bootstrap required, opened a new log
I20260811 02:39:54.371806 23953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:54.371981 23953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:54.372016 23953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b1cef8eac445c58972f4d02640a403, State: Initialized, Role: FOLLOWER
I20260811 02:39:54.372205 23953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:54.372289 23953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:39:54.372355 23953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:39:54.372430 23953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:54.373106 23953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:54.373287 23953 leader_election.cc:302] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b1cef8eac445c58972f4d02640a403; no voters:
I20260811 02:39:54.373508 23953 leader_election.cc:288] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:39:54.373560 23959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:39:54.373842 23959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 LEADER]: Becoming Leader. State: Replica: 40b1cef8eac445c58972f4d02640a403, State: Running, Role: LEADER
I20260811 02:39:54.373912 23953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:39:54.374017 23959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:54.374437 23959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b1cef8eac445c58972f4d02640a403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } } }
I20260811 02:39:54.374572 23959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: This master's current role is: LEADER
I20260811 02:39:54.374730 23959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b1cef8eac445c58972f4d02640a403. Latest consensus state: current_term: 1 leader_uuid: "40b1cef8eac445c58972f4d02640a403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } } }
I20260811 02:39:54.374820 23959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: This master's current role is: LEADER
I20260811 02:39:54.374977 23971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:39:54.375491 23971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:39:54.375836 23953 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 11 02:39:54 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:54.379004 23971 catalog_manager.cc:1347] Generated new cluster ID: c0f53c4d8e864e0f8ecdddacd57895be
I20260811 02:39:54.379112 23971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:39:54.381909 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:54.368534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.2.251.62:42217)
0811 02:39:54.368840 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:54.368854 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:54.368913 (+ 59us) server_negotiation.cc:407] Connection header received
0811 02:39:54.370167 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:54.370181 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:54.370263 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:54.370453 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:54.371405 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:54.372868 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:54.373957 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:54.374605 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:54.374696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:54.377821 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:54.377826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:54.377826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:54.377840 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:54.379816 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:54.380230 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:54.380234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:54.380234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:54.380329 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:54.380516 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:54.380517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:54.380517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:54.380818 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:54.380985 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:54.381494 (+ 509us) server_negotiation.cc:299] Negotiation successful
0811 02:39:54.381619 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":144,"threads_started":1}
I20260811 02:39:54.434731 23971 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:39:54.435678 23971 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:39:54.445436 23971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Generated new TSK 0
I20260811 02:39:54.445855 23971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:39:55.152551 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42217
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:55.258744 24038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:55.258980 24038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:55.259006 24038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:55.260329 24038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:55.260387 24038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:55.260417 24038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:39:55.262166 24038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.251.20:35731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:55.262550 24038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:55.262796 24038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:55.268384 24048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:55.268414 24052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:55.268383 24050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:55.268848 24038 server_base.cc:1034] running on GCE node
I20260811 02:39:55.269075 24038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:55.269327 24038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:55.270481 24038 hybrid_clock.cc:648] HybridClock initialized: now 1786415995270457 us; error 38 us; skew 500 ppm
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:55.273914 24038 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:39:55.274469 24038 webserver.cc:460] Webserver started at http://127.2.251.1:33595/ using document root <none> and password file <none>
I20260811 02:39:55.274726 24038 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:55.274812 24038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:55.274988 24038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:55.291390 24038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "a9b3638d98f64e1dabc0bfc44704dba0"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "d43938ce017e140b289d15de93966a6e"
server_key_iv: "69b5a2c8449b047c1372abd066f5a9d8"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.292006 24038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "a9b3638d98f64e1dabc0bfc44704dba0"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "d43938ce017e140b289d15de93966a6e"
server_key_iv: "69b5a2c8449b047c1372abd066f5a9d8"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.298318 24038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260811 02:39:55.315980 24065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.316269 24038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260811 02:39:55.316383 24038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "a9b3638d98f64e1dabc0bfc44704dba0"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "d43938ce017e140b289d15de93966a6e"
server_key_iv: "69b5a2c8449b047c1372abd066f5a9d8"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.316509 24038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:55.333972 24038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:55.336472 24038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:55.336709 24038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:55.337044 24038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:55.337513 24038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:55.337606 24038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.337749 24038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:55.337832 24038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.344659 24038 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:34607
I20260811 02:39:55.345268 24038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:39:55.353107 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 24038
I20260811 02:39:55.353229 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:39:55.353447 3052 external_mini_cluster.cc:1428] Setting key fe1312e42b543e2102b73ff4b9bc4044
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:55.348289 24254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:34607 every 8 connection(s)
I20260811 02:39:55.361395 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:55.349325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42217 (local address 127.2.251.1:48189)
0811 02:39:55.349680 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:55.349753 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:55.350456 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:55.350885 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:55.350891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:55.351180 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:55.351496 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:55.351503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.352579 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.352580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:55.354101 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:55.354105 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.354666 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.354668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:55.354809 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:55.355448 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:55.355460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:55.357207 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:55.359600 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:55.359608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:55.359612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:55.359938 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:55.360161 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:55.360161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:55.360162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:55.360300 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:55.360606 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:55.360614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:55.360821 (+ 207us) client_negotiation.cc:769] Sending connection context
0811 02:39:55.360935 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 02:39:55.361131 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260811 02:39:55.364746 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:55.349102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:48189 (local address 127.2.251.62:42217)
0811 02:39:55.349611 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:55.349614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:55.350608 (+ 994us) server_negotiation.cc:407] Connection header received
0811 02:39:55.350643 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:55.350644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:55.350660 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:55.350726 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:55.351615 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.352462 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.354204 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.354502 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.354570 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:55.357335 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:55.357338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:55.357338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:55.357346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:55.359478 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:55.360028 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:55.360029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:55.360029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:55.360098 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:55.360363 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:55.360364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:55.360364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:55.360522 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:55.360578 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:55.364528 (+ 3950us) server_negotiation.cc:299] Negotiation successful
0811 02:39:55.364651 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":100,"threads_started":1}
I20260811 02:39:55.365687 24256 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42217
I20260811 02:39:55.365828 24256 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:55.366051 24256 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:55.366750 23888 ts_manager.cc:194] Registered new tserver with Master: a9b3638d98f64e1dabc0bfc44704dba0 (127.2.251.1:34607)
I20260811 02:39:55.367700 23888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:48189
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:39:55.409443 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42217
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 02:39:55.506002 24266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:55.506326 24266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:55.506371 24266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:55.508260 24266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:55.508337 24266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:55.508379 24266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:39:55.510767 24266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.251.20:35731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:55.511081 24266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:55.511379 24266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:55.517361 24272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:55.517402 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:55.517526 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:55.517889 24266 server_base.cc:1034] running on GCE node
I20260811 02:39:55.518121 24266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:55.518373 24266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:55.519541 24266 hybrid_clock.cc:648] HybridClock initialized: now 1786415995519526 us; error 27 us; skew 500 ppm
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:55.522456 24266 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:39:55.522969 24266 webserver.cc:460] Webserver started at http://127.2.251.2:44867/ using document root <none> and password file <none>
I20260811 02:39:55.523222 24266 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:55.523272 24266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:55.523414 24266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:55.538957 24266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "493c01bd4b7f43bab6dae08ce4767c5a"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "83aaf89d8c0f7e24c0bfd9978bc0e8da"
server_key_iv: "0ed15212569f10ff4f42efb197789643"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.539559 24266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "493c01bd4b7f43bab6dae08ce4767c5a"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "83aaf89d8c0f7e24c0bfd9978bc0e8da"
server_key_iv: "0ed15212569f10ff4f42efb197789643"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.541560 24266 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260811 02:39:55.565876 24291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.566167 24266 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260811 02:39:55.566404 24266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "493c01bd4b7f43bab6dae08ce4767c5a"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "83aaf89d8c0f7e24c0bfd9978bc0e8da"
server_key_iv: "0ed15212569f10ff4f42efb197789643"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.566567 24266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:55.588254 24266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:55.590811 24266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:55.590977 24266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:55.591241 24266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:55.591656 24266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:55.591709 24266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.591746 24266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:55.591763 24266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.598364 24266 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:36281
I20260811 02:39:55.598877 24266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:39:55.607316 24505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:36281 every 8 connection(s)
I20260811 02:39:55.608132 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 24266
I20260811 02:39:55.608268 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:39:55.608495 3052 external_mini_cluster.cc:1428] Setting key a980d2b7a625540eea95f3bda1eac2f0
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:55.622875 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:55.608680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:58037 (local address 127.2.251.62:42217)
0811 02:39:55.608797 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:55.608800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:55.611598 (+ 2798us) server_negotiation.cc:407] Connection header received
0811 02:39:55.612395 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:55.612396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:55.612411 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:55.612443 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:55.613405 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.614235 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.615799 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.616152 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.616197 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:55.619023 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:55.619025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:55.619026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:55.619034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:55.620884 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:55.621528 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:55.621530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:55.621530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:55.621598 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:55.621925 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:55.621926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:55.621926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:55.622100 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:55.622147 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:55.622646 (+ 499us) server_negotiation.cc:299] Negotiation successful
0811 02:39:55.622759 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 02:39:55.623492 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:55.608109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42217 (local address 127.2.251.2:58037)
0811 02:39:55.611414 (+ 3305us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:55.611492 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:55.612254 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:55.612666 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:55.612671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:55.612970 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:55.613289 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:55.613295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.614372 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.614373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:55.615689 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:55.615692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.616321 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.616322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:55.616480 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:55.617120 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:55.617132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:55.618877 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:55.621018 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:55.621028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:55.621032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:55.621435 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:55.621663 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:55.621664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:55.621665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:55.621837 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:55.622211 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:55.622219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:55.622480 (+ 261us) client_negotiation.cc:769] Sending connection context
0811 02:39:55.623040 (+ 560us) client_negotiation.cc:240] Negotiation successful
0811 02:39:55.623232 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3235,"thread_start_us":143,"threads_started":1}
I20260811 02:39:55.623983 24507 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42217
I20260811 02:39:55.624100 24507 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:55.624322 24507 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:55.624722 23888 ts_manager.cc:194] Registered new tserver with Master: 493c01bd4b7f43bab6dae08ce4767c5a (127.2.251.2:36281)
I20260811 02:39:55.625247 23888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:58037
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:39:55.666474 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42217
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:55.790472 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:55.790707 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:55.790733 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:55.797051 24514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:55.797158 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:55.797204 24514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:39:55.800236 24514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.251.20:35731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:55.800594 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:55.800879 24514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:55.807607 24524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:55.807686 24523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:55.808207 24529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:55.808624 24514 server_base.cc:1034] running on GCE node
I20260811 02:39:55.808877 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:55.809181 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:55.810509 24514 hybrid_clock.cc:648] HybridClock initialized: now 1786415995810485 us; error 37 us; skew 500 ppm
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:55.814052 24514 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:39:55.814687 24514 webserver.cc:460] Webserver started at http://127.2.251.3:37759/ using document root <none> and password file <none>
I20260811 02:39:55.814973 24514 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:55.815037 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:39:55.815187 24514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:55.838651 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "e6cb55995a8a45a29a9b797d308b41de"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "2d117cf9198ba04a0057a926269f16b0"
server_key_iv: "ec57bc1c6fc19aa1442e1a7325337c39"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.839155 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "e6cb55995a8a45a29a9b797d308b41de"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "2d117cf9198ba04a0057a926269f16b0"
server_key_iv: "ec57bc1c6fc19aa1442e1a7325337c39"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.845649 24514 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260811 02:39:55.861580 24541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.862114 24514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 02:39:55.862216 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "e6cb55995a8a45a29a9b797d308b41de"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "2d117cf9198ba04a0057a926269f16b0"
server_key_iv: "ec57bc1c6fc19aa1442e1a7325337c39"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:55.862317 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:55.913640 24514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:55.916330 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:55.916505 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:55.916743 24514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:55.917143 24514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:55.917187 24514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.917215 24514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:55.917240 24514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:55.924116 24514 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:32899
I20260811 02:39:55.924541 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:32899 every 8 connection(s)
I20260811 02:39:55.924644 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:39:55.932250 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 24514
I20260811 02:39:55.932377 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:39:55.932581 3052 external_mini_cluster.cc:1428] Setting key 073b56d333a18a602a7d830c0cb53c9a
Aug 11 02:39:55 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:55.942260 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:55.925820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:50667 (local address 127.2.251.62:42217)
0811 02:39:55.925873 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:55.925875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:55.931762 (+ 5887us) server_negotiation.cc:407] Connection header received
0811 02:39:55.932491 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:55.932493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:55.932508 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:55.932538 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:55.933421 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.934214 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.935752 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.936056 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.936103 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:55.938898 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:55.938899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:55.938900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:55.938919 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:55.940552 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:55.941125 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:55.941126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:55.941126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:55.941187 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:55.941476 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:55.941476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:55.941477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:55.941616 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:55.941683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:55.942078 (+ 395us) server_negotiation.cc:299] Negotiation successful
0811 02:39:55.942164 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:39:55.943004 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:55.925495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42217 (local address 127.2.251.3:50667)
0811 02:39:55.931545 (+ 6050us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:55.931601 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:55.932345 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:55.932748 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:55.932753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:55.933024 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:55.933331 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:55.933338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.934333 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.934334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:55.935637 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:55.935640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:55.936219 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:55.936220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:55.936369 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:55.937062 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:55.937074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:55.938774 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:55.940661 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:55.940671 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:55.940674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:55.941044 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:55.941255 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:55.941255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:55.941256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:55.941411 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:55.941682 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:55.941689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:55.941923 (+ 234us) client_negotiation.cc:769] Sending connection context
0811 02:39:55.942052 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 02:39:55.942226 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5994,"thread_start_us":98,"threads_started":1}
I20260811 02:39:55.943605 24736 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42217
I20260811 02:39:55.943720 24736 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:55.943953 24736 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:55.944411 23888 ts_manager.cc:194] Registered new tserver with Master: e6cb55995a8a45a29a9b797d308b41de (127.2.251.3:32899)
I20260811 02:39:55.944902 23888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:50667
I20260811 02:39:55.945184 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:39:55.946027 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 24038
I20260811 02:39:55.966696 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 24266
I20260811 02:39:55.987958 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 24514
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415995, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:56.036715 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 23831
I20260811 02:39:56.059106 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42217
--webserver_interface=127.2.251.62
--webserver_port=36549
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:42217
--ranger_config_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:39:56.146073 24742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:56.146310 24742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:56.146332 24742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:56.147485 24742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:39:56.147535 24742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:56.147548 24742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:56.147560 24742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:39:56.147571 24742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:39:56.149199 24742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:42217
--ranger_config_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=36549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:56.149454 24742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:56.149679 24742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:39:56.154814 24742 server_base.cc:1034] running on GCE node
W20260811 02:39:56.154923 24757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.154876 24753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.154879 24749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:56.155339 24742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:56.155663 24742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:56.156821 24742 hybrid_clock.cc:648] HybridClock initialized: now 1786415996156808 us; error 31 us; skew 500 ppm
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:56.159734 24742 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:39:56.160215 24742 webserver.cc:460] Webserver started at http://127.2.251.62:36549/ using document root <none> and password file <none>
I20260811 02:39:56.160463 24742 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:56.160523 24742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:56.230553 24742 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:39:56.231587 24772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.231823 24742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:39:56.231927 24742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "40b1cef8eac445c58972f4d02640a403"
format_stamp: "Formatted at 2026-08-11 02:39:54 on dist-test-slave-cvwc"
server_key: "26200fe0c11fafa013c020ff77c76e32"
server_key_iv: "c636b33470539c1076d88993020278ea"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:56.232460 24742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:56.259601 24742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:56.261566 24742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:56.261698 24742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:56.266088 24742 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:42217
I20260811 02:39:56.266279 24860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:42217 every 8 connection(s)
I20260811 02:39:56.266606 24742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:39:56.266752 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 24742
I20260811 02:39:56.267181 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:34607
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:42217
--webserver_port=33595
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:39:56.268360 24861 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:39:56.269261 24861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Bootstrap starting.
I20260811 02:39:56.277520 24861 log.cc:824] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:56.280390 24861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:39:56.281030 24861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Bootstrap complete.
I20260811 02:39:56.283900 24861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:56.284206 24861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b1cef8eac445c58972f4d02640a403, State: Initialized, Role: FOLLOWER
I20260811 02:39:56.284382 24861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:56.284466 24861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:39:56.284516 24861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:39:56.284564 24861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 1 FOLLOWER]: Advancing to term 2
I20260811 02:39:56.285797 24861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:56.285931 24861 leader_election.cc:302] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b1cef8eac445c58972f4d02640a403; no voters:
I20260811 02:39:56.286145 24861 leader_election.cc:288] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 02:39:56.286199 24867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 02:39:56.286325 24867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [term 2 LEADER]: Becoming Leader. State: Replica: 40b1cef8eac445c58972f4d02640a403, State: Running, Role: LEADER
I20260811 02:39:56.286459 24861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:39:56.286459 24867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } }
I20260811 02:39:56.286813 24873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40b1cef8eac445c58972f4d02640a403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } } }
I20260811 02:39:56.286983 24873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: This master's current role is: LEADER
I20260811 02:39:56.286815 24867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b1cef8eac445c58972f4d02640a403. Latest consensus state: current_term: 2 leader_uuid: "40b1cef8eac445c58972f4d02640a403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1cef8eac445c58972f4d02640a403" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 42217 } } }
I20260811 02:39:56.287258 24867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403 [sys.catalog]: This master's current role is: LEADER
I20260811 02:39:56.287262 24885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:39:56.287889 24885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:39:56.288049 24885 catalog_manager.cc:1259] Loaded cluster ID: c0f53c4d8e864e0f8ecdddacd57895be
I20260811 02:39:56.288084 24885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:39:56.288374 24861 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 02:39:56.289269 24885 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:39:56.289486 24885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Loaded TSK: 0
I20260811 02:39:56.289743 24885 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 02:39:56.382786 24865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:56.383101 24865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:56.383145 24865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:56.385053 24865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:56.385133 24865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:56.385181 24865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:39:56.387900 24865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:34607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=33595
--tserver_master_addrs=127.2.251.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:56.388247 24865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:56.388677 24865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:56.398061 24936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.398061 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.398175 24930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:56.398721 24865 server_base.cc:1034] running on GCE node
I20260811 02:39:56.398959 24865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:56.399273 24865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:56.400451 24865 hybrid_clock.cc:648] HybridClock initialized: now 1786415996400428 us; error 48 us; skew 500 ppm
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:56.403970 24865 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:39:56.404608 24865 webserver.cc:460] Webserver started at http://127.2.251.1:33595/ using document root <none> and password file <none>
I20260811 02:39:56.404944 24865 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:56.405066 24865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:56.431836 24865 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260811 02:39:56.433091 24948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.433301 24865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:39:56.433393 24865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "a9b3638d98f64e1dabc0bfc44704dba0"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "d43938ce017e140b289d15de93966a6e"
server_key_iv: "69b5a2c8449b047c1372abd066f5a9d8"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:56.433976 24865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:56.457476 24865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:56.459762 24865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:56.459952 24865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:56.460232 24865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:56.460676 24865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:56.460738 24865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.460772 24865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:56.460794 24865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.468858 24865 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:34607
I20260811 02:39:56.469447 24865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:39:56.469965 25149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:34607 every 8 connection(s)
I20260811 02:39:56.474776 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 24865
I20260811 02:39:56.475116 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:36281
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:42217
--webserver_port=44867
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:56.482326 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.470363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42217 (local address 127.2.251.1:45107)
0811 02:39:56.470545 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:56.470596 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:56.471269 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:56.472170 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:56.472176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:56.472705 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:56.472958 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:56.472968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.474233 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.474235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:56.475149 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:56.475153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.475858 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.475859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:56.475999 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.476588 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:56.476600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:56.478419 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:56.480376 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:56.480384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:56.480386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:56.480680 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:56.480935 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:56.480936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:56.480937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:56.481045 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:56.481546 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:56.481553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:56.481748 (+ 195us) client_negotiation.cc:769] Sending connection context
0811 02:39:56.481866 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 02:39:56.482063 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260811 02:39:56.482903 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.470565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:45107 (local address 127.2.251.62:42217)
0811 02:39:56.471518 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:56.471527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:56.471563 (+ 36us) server_negotiation.cc:407] Connection header received
0811 02:39:56.471727 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:56.471734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:56.471793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:56.471936 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:56.473127 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.474116 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.475265 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.475738 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.475845 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.478614 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:56.478620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:56.478620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:56.478632 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:56.480267 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.480766 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.480770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.480771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.480869 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.481107 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.481108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.481109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.481373 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:56.481520 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:56.482548 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0811 02:39:56.482697 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":82,"threads_started":1}
I20260811 02:39:56.483592 25152 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42217
I20260811 02:39:56.483719 25152 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.484002 25152 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
W20260811 02:39:56.485108 25152 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:42217 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 02:39:56.485185 25152 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.485267 25152 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:56.485589 25152 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.485678 25152 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
W20260811 02:39:56.485999 25152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:39:56.606827 25160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:56.607174 25160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:56.607223 25160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:56.609197 25160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:56.609287 25160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:56.609331 25160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:39:56.612018 25160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=44867
--tserver_master_addrs=127.2.251.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:56.612350 25160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:56.612680 25160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:56.618700 25170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.618701 25172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.619148 25168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:56.619338 25160 server_base.cc:1034] running on GCE node
I20260811 02:39:56.619590 25160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:56.619899 25160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:56.621112 25160 hybrid_clock.cc:648] HybridClock initialized: now 1786415996621080 us; error 45 us; skew 500 ppm
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:56.624795 25160 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:39:56.625306 25160 webserver.cc:460] Webserver started at http://127.2.251.2:44867/ using document root <none> and password file <none>
I20260811 02:39:56.625537 25160 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:56.625607 25160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:56.650421 25160 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260811 02:39:56.651845 25188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.652032 25160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:39:56.652143 25160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "493c01bd4b7f43bab6dae08ce4767c5a"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "83aaf89d8c0f7e24c0bfd9978bc0e8da"
server_key_iv: "0ed15212569f10ff4f42efb197789643"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:56.652583 25160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:56.705201 25160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:56.708009 25160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:56.708199 25160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:56.708503 25160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:56.708937 25160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:56.708992 25160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.709022 25160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:56.709045 25160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.717113 25160 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:36281
I20260811 02:39:56.717186 25380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:36281 every 8 connection(s)
I20260811 02:39:56.717656 25160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:56.725088 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 25160
I20260811 02:39:56.725384 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.251.20:35731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:32899
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:42217
--webserver_port=37759
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:39:56.729782 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.718648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:51763 (local address 127.2.251.62:42217)
0811 02:39:56.718695 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:56.718697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:56.718973 (+ 276us) server_negotiation.cc:407] Connection header received
0811 02:39:56.719514 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:56.719516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:56.719528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:56.719580 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:56.720165 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.720701 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.721686 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.721939 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.721994 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.724113 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:56.724115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:56.724115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:56.724124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:56.728031 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.728601 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.728602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.728603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.728668 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.728970 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.728971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.728971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.729139 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:56.729184 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:56.729563 (+ 379us) server_negotiation.cc:299] Negotiation successful
0811 02:39:56.729655 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 02:39:56.729972 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.718659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42217 (local address 127.2.251.2:51763)
0811 02:39:56.718836 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:56.718875 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:56.719396 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:56.719654 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:56.719658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:56.719851 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:56.720077 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:56.720082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.720789 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.720790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:56.721601 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:56.721603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.722040 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.722042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:56.722151 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.722679 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:56.722689 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:56.724016 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:56.728134 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:56.728147 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:56.728151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:56.728504 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:56.728735 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:56.728735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:56.728736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:56.728884 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:56.729234 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:56.729242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:56.729428 (+ 186us) client_negotiation.cc:769] Sending connection context
0811 02:39:56.729527 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 02:39:56.729693 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260811 02:39:56.730636 25381 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42217
I20260811 02:39:56.730738 25381 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.730934 25381 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
W20260811 02:39:56.731698 25381 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:42217 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 02:39:56.731787 25381 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.731863 25381 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:56.732245 25381 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.732331 25381 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
W20260811 02:39:56.732653 25381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:39:56.853602 25385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:39:56.854007 25385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:39:56.854050 25385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:39:56.856026 25385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:39:56.856096 25385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:39:56.856139 25385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:39:56.858721 25385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.251.20:35731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:32899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=37759
--tserver_master_addrs=127.2.251.62:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:39:56.859061 25385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:39:56.859355 25385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:39:56.864898 25399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.864902 25395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:39:56.865264 25397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:39:56.865290 25385 server_base.cc:1034] running on GCE node
I20260811 02:39:56.865549 25385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:39:56.865878 25385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:39:56.867053 25385 hybrid_clock.cc:648] HybridClock initialized: now 1786415996867023 us; error 44 us; skew 500 ppm
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:56.870316 25385 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:39:56.871225 25385 webserver.cc:460] Webserver started at http://127.2.251.3:37759/ using document root <none> and password file <none>
I20260811 02:39:56.871522 25385 fs_manager.cc:359] Metadata directory not provided
I20260811 02:39:56.871603 25385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:56.897998 25385 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260811 02:39:56.899297 25413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.899538 25385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:39:56.899668 25385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "e6cb55995a8a45a29a9b797d308b41de"
format_stamp: "Formatted at 2026-08-11 02:39:55 on dist-test-slave-cvwc"
server_key: "2d117cf9198ba04a0057a926269f16b0"
server_key_iv: "ec57bc1c6fc19aa1442e1a7325337c39"
server_key_version: "kuduclusterkey@0"
I20260811 02:39:56.900202 25385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:39:56.930733 25385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:39:56.933372 25385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:39:56.933569 25385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:39:56.936343 25385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:39:56.936859 25385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:39:56.936928 25385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.936959 25385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:39:56.937021 25385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:39:56.947194 25385 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:32899
I20260811 02:39:56.947528 25603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:32899 every 8 connection(s)
I20260811 02:39:56.947768 25385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:39:56.949354 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 25385
Aug 11 02:39:56 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415996, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:39:56.968690 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.949794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.2.251.62:42217)
0811 02:39:56.950841 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:56.950843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:56.950850 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:39:56.950874 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:56.950874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:56.950884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:56.950958 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:56.951778 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.952555 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.953687 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.959379 (+ 5692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.959491 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.962617 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:56.962619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:56.962620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:56.962629 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:56.966295 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.967350 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.967352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.967352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.967439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.967998 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.968000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.968000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.968193 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:56.968426 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:56.968477 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 02:39:56.968581 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":75,"threads_started":1}
I20260811 02:39:56.969929 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.948941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:42377 (local address 127.2.251.62:42217)
0811 02:39:56.949034 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:56.949036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:56.949317 (+ 281us) server_negotiation.cc:407] Connection header received
0811 02:39:56.950544 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:56.950546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:56.950560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:56.950636 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:56.951472 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.952471 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.954006 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.954369 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.954421 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.963150 (+ 8729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:56.963151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:56.963152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:56.963159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:56.966401 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.967385 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.967386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.967386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.967452 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:56.968008 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:56.968009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:56.968010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:56.968210 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:56.968301 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:56.969646 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0811 02:39:56.969837 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":273,"server-negotiator.queue_time_us":54}
I20260811 02:39:56.970806 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:56.948630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:42217 (local address 127.2.251.3:42377)
0811 02:39:56.949178 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:56.949228 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:56.950408 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:56.950786 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:56.950792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:56.951056 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:39:56.951354 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:56.951360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.952590 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.952591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:56.953908 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:56.953911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:56.954542 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:56.954543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:56.954693 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:56.955405 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:39:56.955416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:39:56.963004 (+ 7588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:39:56.966520 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:56.966537 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:56.966540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:56.966925 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:56.967529 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:39:56.967530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:39:56.967530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:39:56.967717 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:39:56.968759 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:39:56.968770 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:39:56.969370 (+ 600us) client_negotiation.cc:769] Sending connection context
0811 02:39:56.970184 (+ 814us) client_negotiation.cc:240] Negotiation successful
0811 02:39:56.970400 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":88,"threads_started":1}
I20260811 02:39:56.971310 25604 heartbeater.cc:347] Connected to a master server at 127.2.251.62:42217
I20260811 02:39:56.971438 25604 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.971659 25604 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
W20260811 02:39:56.972545 25604 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:42217 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 02:39:56.972633 25604 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.972707 25604 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:56.973095 25604 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:56.973177 25604 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
W20260811 02:39:56.973495 25604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 02:39:57.486198 25152 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:57.486402 25152 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:57.486922 24799 ts_manager.cc:194] Registered new tserver with Master: a9b3638d98f64e1dabc0bfc44704dba0 (127.2.251.1:34607)
I20260811 02:39:57.488062 24799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:45107
Aug 11 02:39:57 dist-test-slave-cvwc krb5kdc[23050](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786415997, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:39:57.732892 25381 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:57.733083 25381 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:57.733613 24799 ts_manager.cc:194] Registered new tserver with Master: 493c01bd4b7f43bab6dae08ce4767c5a (127.2.251.2:36281)
I20260811 02:39:57.734179 24799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:51763
I20260811 02:39:57.973698 25604 heartbeater.cc:464] Registering TS with master...
I20260811 02:39:57.973922 25604 heartbeater.cc:511] Master 127.2.251.62:42217 requested a full tablet report, sending...
I20260811 02:39:57.974263 24799 ts_manager.cc:194] Registered new tserver with Master: e6cb55995a8a45a29a9b797d308b41de (127.2.251.3:32899)
I20260811 02:39:57.974673 24799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:42377
Aug 11 02:39:57 dist-test-slave-cvwc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786415997, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for HTTP/127.2.251.20@KRBTEST.COM
I20260811 02:39:57.980412 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:39:57.991032 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:57.981413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.2.251.62:42217)
0811 02:39:57.981691 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:57.981694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:57.981900 (+ 206us) server_negotiation.cc:407] Connection header received
0811 02:39:57.981926 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:57.981927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:57.981939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:57.981995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:39:57.982706 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:57.983416 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:57.985492 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:57.985844 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:57.985934 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:57.987599 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:39:57.987600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:39:57.987600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:39:57.987609 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:39:57.989418 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:57.989841 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:57.989843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:57.989843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:57.989909 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:39:57.990110 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:39:57.990111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:39:57.990111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:39:57.990250 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:39:57.990343 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:57.990838 (+ 495us) server_negotiation.cc:299] Negotiation successful
0811 02:39:57.990926 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":86,"threads_started":1}
I20260811 02:39:57.993346 24799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:39:57.995831 24799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:39:58.007474 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.001183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:32899 (local address 127.0.0.1:43942)
0811 02:39:58.001977 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:58.002021 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:58.002093 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:58.002419 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:58.002422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:58.002425 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:58.002505 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.002507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.003736 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.003737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.005412 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.005416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.006981 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.006984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.007138 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.007162 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 02:39:58.007264 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 02:39:58.007356 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":81,"threads_started":1}
I20260811 02:39:58.008638 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.001465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.2.251.3:32899)
0811 02:39:58.001896 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:58.001902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:58.002176 (+ 274us) server_negotiation.cc:407] Connection header received
0811 02:39:58.002216 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:58.002218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:58.002234 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:58.002312 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:58.002586 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.003633 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.005534 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.006690 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.006831 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.006934 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:58.008463 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0811 02:39:58.008535 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":67,"threads_started":1}
I20260811 02:39:58.009791 25475 tablet_service.cc:1467] Processing CreateTablet for tablet 80da4250bcd04e8bad956d133211b00f (DEFAULT_TABLE table=test-table [id=d95cb67978cd4ca8a8af14723a6b5a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:39:58.010234 25475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80da4250bcd04e8bad956d133211b00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:58.017827 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.001211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:36281 (local address 127.0.0.1:52126)
0811 02:39:58.004432 (+ 3221us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:58.004445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:58.004498 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:58.010630 (+ 6132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:58.010631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:58.010634 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:58.010681 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.010683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.015522 (+ 4839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.015524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.016512 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.016516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.017482 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.017483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.017580 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.017585 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:39:58.017651 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 02:39:58.017702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3200,"thread_start_us":123,"threads_started":1}
I20260811 02:39:58.018107 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.001669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.2.251.2:36281)
0811 02:39:58.009340 (+ 7671us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:58.009346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:58.009360 (+ 14us) server_negotiation.cc:407] Connection header received
0811 02:39:58.009401 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:58.009403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:58.009420 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:58.010756 (+ 1336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:58.010847 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.011906 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.016631 (+ 4725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.017379 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.017474 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.017550 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:58.017975 (+ 425us) server_negotiation.cc:299] Negotiation successful
0811 02:39:58.018028 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7615,"thread_start_us":84,"threads_started":1}
I20260811 02:39:58.018730 25674 tablet_bootstrap.cc:492] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Bootstrap starting.
I20260811 02:39:58.018961 25269 tablet_service.cc:1467] Processing CreateTablet for tablet 80da4250bcd04e8bad956d133211b00f (DEFAULT_TABLE table=test-table [id=d95cb67978cd4ca8a8af14723a6b5a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:39:58.019352 25269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80da4250bcd04e8bad956d133211b00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:58.019486 25674 tablet_bootstrap.cc:654] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:58.019793 25674 log.cc:824] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:58.020607 25674 tablet_bootstrap.cc:492] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: No bootstrap required, opened a new log
I20260811 02:39:58.020673 25674 ts_tablet_manager.cc:1397] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:39:58.023352 25674 raft_consensus.cc:348] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.023520 25674 raft_consensus.cc:374] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:58.023551 25674 raft_consensus.cc:729] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cb55995a8a45a29a9b797d308b41de, State: Initialized, Role: FOLLOWER
I20260811 02:39:58.023763 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.001468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34607 (local address 127.0.0.1:60796)
0811 02:39:58.008092 (+ 6624us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:58.008108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:58.008185 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:58.010246 (+ 2061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:58.010248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:58.010250 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:58.010436 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.010440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.013144 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.013145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.014547 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.014551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.017619 (+ 3068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.017620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.017697 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.017702 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:39:58.017780 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 02:39:58.017878 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6595,"thread_start_us":71,"threads_started":1}
I20260811 02:39:58.024267 25674 consensus_queue.cc:260] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.024427 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.001562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.2.251.1:34607)
0811 02:39:58.009892 (+ 8330us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:58.009897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:58.009909 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:39:58.009944 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:58.009946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:58.009962 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:58.010139 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:58.012019 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.013032 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.014658 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.015390 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.024107 (+ 8717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.024218 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:58.024251 (+ 33us) server_negotiation.cc:299] Negotiation successful
0811 02:39:58.024307 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7975,"thread_start_us":61,"threads_started":1}
I20260811 02:39:58.024592 25674 ts_tablet_manager.cc:1428] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 02:39:58.024716 25604 heartbeater.cc:503] Master 127.2.251.62:42217 was elected leader, sending a full tablet report...
I20260811 02:39:58.025188 25011 tablet_service.cc:1467] Processing CreateTablet for tablet 80da4250bcd04e8bad956d133211b00f (DEFAULT_TABLE table=test-table [id=d95cb67978cd4ca8a8af14723a6b5a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:39:58.025549 25011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80da4250bcd04e8bad956d133211b00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:39:58.025820 25679 tablet_bootstrap.cc:492] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Bootstrap starting.
I20260811 02:39:58.026515 25679 tablet_bootstrap.cc:654] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:58.026844 25679 log.cc:824] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:58.028198 25679 tablet_bootstrap.cc:492] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: No bootstrap required, opened a new log
I20260811 02:39:58.028285 25679 ts_tablet_manager.cc:1397] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 02:39:58.030339 25691 tablet_bootstrap.cc:492] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0: Bootstrap starting.
I20260811 02:39:58.031188 25691 tablet_bootstrap.cc:654] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0: Neither blocks nor log segments found. Creating new log.
I20260811 02:39:58.031575 25691 log.cc:824] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0: Log is configured to *not* fsync() on all Append() calls
I20260811 02:39:58.031500 25679 raft_consensus.cc:348] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.031672 25679 raft_consensus.cc:374] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:58.031718 25679 raft_consensus.cc:729] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493c01bd4b7f43bab6dae08ce4767c5a, State: Initialized, Role: FOLLOWER
I20260811 02:39:58.031895 25679 consensus_queue.cc:260] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.032183 25679 ts_tablet_manager.cc:1428] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260811 02:39:58.032312 25381 heartbeater.cc:503] Master 127.2.251.62:42217 was elected leader, sending a full tablet report...
I20260811 02:39:58.032773 25691 tablet_bootstrap.cc:492] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0: No bootstrap required, opened a new log
I20260811 02:39:58.032842 25691 ts_tablet_manager.cc:1397] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:39:58.035215 25691 raft_consensus.cc:348] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.035429 25691 raft_consensus.cc:374] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:39:58.035480 25691 raft_consensus.cc:729] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b3638d98f64e1dabc0bfc44704dba0, State: Initialized, Role: FOLLOWER
I20260811 02:39:58.035758 25691 consensus_queue.cc:260] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.036837 25691 ts_tablet_manager.cc:1428] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 02:39:58.036983 25152 heartbeater.cc:503] Master 127.2.251.62:42217 was elected leader, sending a full tablet report...
I20260811 02:39:58.057278 25693 raft_consensus.cc:482] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:39:58.057467 25693 raft_consensus.cc:504] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.057971 25693 leader_election.cc:288] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6cb55995a8a45a29a9b797d308b41de (127.2.251.3:32899), a9b3638d98f64e1dabc0bfc44704dba0 (127.2.251.1:34607)
I20260811 02:39:58.063542 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.058534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:51267 (local address 127.2.251.3:32899)
0811 02:39:58.058580 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:58.058582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:58.059076 (+ 494us) server_negotiation.cc:407] Connection header received
0811 02:39:58.059183 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:58.059185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:58.059198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:58.059226 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:58.059472 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.060171 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.062041 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.062638 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.062705 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.062717 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:58.063414 (+ 697us) server_negotiation.cc:299] Negotiation successful
0811 02:39:58.063464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:39:58.063822 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.058254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:32899 (local address 127.2.251.2:51267)
0811 02:39:58.059009 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:58.059021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:58.059107 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:58.059298 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:58.059299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:58.059301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:58.059395 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.059397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.060279 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.060280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.061931 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.061935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.062913 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.062915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.063319 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.063327 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 02:39:58.063678 (+ 351us) client_negotiation.cc:240] Negotiation successful
0811 02:39:58.063725 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":126,"threads_started":1}
I20260811 02:39:58.064168 25517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da4250bcd04e8bad956d133211b00f" candidate_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6cb55995a8a45a29a9b797d308b41de" is_pre_election: true
I20260811 02:39:58.064368 25517 raft_consensus.cc:2454] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493c01bd4b7f43bab6dae08ce4767c5a in term 0.
I20260811 02:39:58.064633 25190 leader_election.cc:302] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493c01bd4b7f43bab6dae08ce4767c5a, e6cb55995a8a45a29a9b797d308b41de; no voters:
I20260811 02:39:58.064785 25693 raft_consensus.cc:2781] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:39:58.064836 25693 raft_consensus.cc:482] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:39:58.064862 25693 raft_consensus.cc:3037] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:58.065869 25693 raft_consensus.cc:504] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.066042 25693 leader_election.cc:288] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [CANDIDATE]: Term 1 election: Requested vote from peers e6cb55995a8a45a29a9b797d308b41de (127.2.251.3:32899), a9b3638d98f64e1dabc0bfc44704dba0 (127.2.251.1:34607)
I20260811 02:39:58.066260 25517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da4250bcd04e8bad956d133211b00f" candidate_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6cb55995a8a45a29a9b797d308b41de"
I20260811 02:39:58.066357 25517 raft_consensus.cc:3037] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:58.067271 25517 raft_consensus.cc:2454] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493c01bd4b7f43bab6dae08ce4767c5a in term 1.
I20260811 02:39:58.067481 25190 leader_election.cc:302] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493c01bd4b7f43bab6dae08ce4767c5a, e6cb55995a8a45a29a9b797d308b41de; no voters:
I20260811 02:39:58.067577 25693 raft_consensus.cc:2781] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:39:58.067725 25693 raft_consensus.cc:686] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 1 LEADER]: Becoming Leader. State: Replica: 493c01bd4b7f43bab6dae08ce4767c5a, State: Running, Role: LEADER
I20260811 02:39:58.067854 25693 consensus_queue.cc:237] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } }
I20260811 02:39:58.068091 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.058891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:37801 (local address 127.2.251.1:34607)
0811 02:39:58.058930 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:58.058932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:58.064099 (+ 5167us) server_negotiation.cc:407] Connection header received
0811 02:39:58.064197 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:58.064198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:58.064212 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:58.064239 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:39:58.064412 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.065123 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.067092 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.067656 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.067701 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.067712 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:58.067973 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 02:39:58.068020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:39:58.068360 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.058701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34607 (local address 127.2.251.2:37801)
0811 02:39:58.064047 (+ 5346us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:58.064057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:39:58.064128 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:39:58.064296 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:39:58.064297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:39:58.064308 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:39:58.064341 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.064343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.065217 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.065218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.066987 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:39:58.066991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.067782 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.067783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:39:58.067840 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.067843 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:39:58.068244 (+ 401us) client_negotiation.cc:240] Negotiation successful
0811 02:39:58.068284 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5332,"thread_start_us":90,"threads_started":1}
I20260811 02:39:58.068678 25039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da4250bcd04e8bad956d133211b00f" candidate_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" is_pre_election: true
I20260811 02:39:58.068837 25039 raft_consensus.cc:2454] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493c01bd4b7f43bab6dae08ce4767c5a in term 0.
I20260811 02:39:58.068766 24799 catalog_manager.cc:5495] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 493c01bd4b7f43bab6dae08ce4767c5a (127.2.251.2). New cstate: current_term: 1 leader_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 } health_report { overall_health: UNKNOWN } } }
I20260811 02:39:58.068967 25070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80da4250bcd04e8bad956d133211b00f" candidate_uuid: "493c01bd4b7f43bab6dae08ce4767c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b3638d98f64e1dabc0bfc44704dba0"
I20260811 02:39:58.069023 25070 raft_consensus.cc:3037] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:39:58.070438 25070 raft_consensus.cc:2454] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493c01bd4b7f43bab6dae08ce4767c5a in term 1.
I20260811 02:39:58.088649 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.083115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.2.251.2:36281)
0811 02:39:58.083187 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 02:39:58.083188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:39:58.083195 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:39:58.083236 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:39:58.083236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:39:58.083259 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:39:58.083336 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:39:58.083524 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.084261 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.085388 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:39:58.086298 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:39:58.086816 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:39:58.087593 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:39:58.087825 (+ 232us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:39:58.087926 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 02:39:58.088509 (+ 583us) server_negotiation.cc:299] Negotiation successful
0811 02:39:58.088542 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 02:39:58.091073 25070 raft_consensus.cc:1261] T 80da4250bcd04e8bad956d133211b00f P a9b3638d98f64e1dabc0bfc44704dba0 [term 1 FOLLOWER]: Refusing update from remote peer 493c01bd4b7f43bab6dae08ce4767c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:39:58.091495 25700 consensus_queue.cc:1035] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b3638d98f64e1dabc0bfc44704dba0" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:39:58.094285 25517 raft_consensus.cc:1261] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 1 FOLLOWER]: Refusing update from remote peer 493c01bd4b7f43bab6dae08ce4767c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:39:58.094627 25700 consensus_queue.cc:1035] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6cb55995a8a45a29a9b797d308b41de" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:39:58.098099 25703 mvcc.cc:205] Tried to move back new op lower bound from 7317159928177541120 to 7317159928086495232. Current Snapshot: MvccSnapshot[applied={T|T < 7317159928177541120}]
I20260811 02:39:58.099117 25712 mvcc.cc:205] Tried to move back new op lower bound from 7317159928177541120 to 7317159928086495232. Current Snapshot: MvccSnapshot[applied={T|T < 7317159928177541120}]
I20260811 02:39:58.101560 25704 mvcc.cc:205] Tried to move back new op lower bound from 7317159928177541120 to 7317159928086495232. Current Snapshot: MvccSnapshot[applied={T|T < 7317159928177541120}]
I20260811 02:39:58.104589 24799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:39:58.104730 24799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:39:58.105612 24799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40b1cef8eac445c58972f4d02640a403: Sending DeleteTablet for 3 replicas of tablet 80da4250bcd04e8bad956d133211b00f
I20260811 02:39:58.106096 25269 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da4250bcd04e8bad956d133211b00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:39:58 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:52126
I20260811 02:39:58.106348 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da4250bcd04e8bad956d133211b00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:39:58 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:60796
I20260811 02:39:58.106746 25475 tablet_service.cc:1514] Processing DeleteTablet for tablet 80da4250bcd04e8bad956d133211b00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:39:58 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:43942
I20260811 02:39:58.107329 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 24865
I20260811 02:39:58.128556 25723 tablet_replica.cc:330] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: stopping tablet replica
I20260811 02:39:58.128746 25723 raft_consensus.cc:2229] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:39:58.128883 25723 raft_consensus.cc:2258] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:39:58.129074 25720 tablet_replica.cc:330] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: stopping tablet replica
I20260811 02:39:58.129190 25720 raft_consensus.cc:2229] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:39:58.129323 25720 raft_consensus.cc:2258] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:39:58.134723 25723 ts_tablet_manager.cc:1905] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:39:58.135401 25720 ts_tablet_manager.cc:1905] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:39:58.138856 25720 ts_tablet_manager.cc:1918] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:39:58.138957 25720 log.cc:1196] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/80da4250bcd04e8bad956d133211b00f
I20260811 02:39:58.139247 25720 ts_tablet_manager.cc:1939] T 80da4250bcd04e8bad956d133211b00f P 493c01bd4b7f43bab6dae08ce4767c5a: Deleting consensus metadata
I20260811 02:39:58.139750 24774 catalog_manager.cc:4841] TS 493c01bd4b7f43bab6dae08ce4767c5a (127.2.251.2:36281): tablet 80da4250bcd04e8bad956d133211b00f (table test-table [id=d95cb67978cd4ca8a8af14723a6b5a2d]) successfully deleted
I20260811 02:39:58.145920 25723 ts_tablet_manager.cc:1918] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:39:58.146045 25723 log.cc:1196] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/80da4250bcd04e8bad956d133211b00f
I20260811 02:39:58.146348 25723 ts_tablet_manager.cc:1939] T 80da4250bcd04e8bad956d133211b00f P e6cb55995a8a45a29a9b797d308b41de: Deleting consensus metadata
I20260811 02:39:58.146898 24773 catalog_manager.cc:4841] TS e6cb55995a8a45a29a9b797d308b41de (127.2.251.3:32899): tablet 80da4250bcd04e8bad956d133211b00f (table test-table [id=d95cb67978cd4ca8a8af14723a6b5a2d]) successfully deleted
W20260811 02:39:58.147567 24776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:34607 (error 108)
I20260811 02:39:58.150019 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 25160
I20260811 02:39:58.234525 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:39:58.234102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:34607 (local address 127.0.0.1:60798)
0811 02:39:58.234410 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 02:39:58.234473 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272}
W20260811 02:39:58.234709 24776 catalog_manager.cc:4584] TS a9b3638d98f64e1dabc0bfc44704dba0 (127.2.251.1:34607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80da4250bcd04e8bad956d133211b00f: Network error: Client connection negotiation failed: client connection to 127.2.251.1:34607: connect: Connection refused (error 111)
I20260811 02:39:58.240163 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 25385
I20260811 02:39:58.259202 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 24742
I20260811 02:39:58.274466 3052 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 02:39:58.625855 3052 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 02:39:58.625962 3052 mini_ranger.cc:66] Stopping Ranger...
I20260811 02:39:58.993139 3052 mini_ranger.cc:68] Stopped Ranger
2026-08-11 02:39:58.993 UTC [23100] LOG: received smart shutdown request
2026-08-11 02:39:58.998 UTC [23100] LOG: background worker "logical replication launcher" (PID 23117) exited with exit code 1
2026-08-11 02:39:58.999 UTC [23107] LOG: shutting down
2026-08-11 02:39:59.237 UTC [23100] LOG: database system is shut down
2026-08-11T02:39:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53022 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:39:59 dist-test-slave-cvwc krb5kdc[25741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:39:59 dist-test-slave-cvwc krb5kdc[25741](info): set up 2 sockets
Aug 11 02:39:59 dist-test-slave-cvwc krb5kdc[25741](info): commencing operation
krb5kdc: starting...
W20260811 02:40:00.581899 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:40:00 dist-test-slave-cvwc krb5kdc[25741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:40:00.726090 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40397
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40397,127.2.251.61:37633,127.2.251.60:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:00.815382 25763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:00.815661 25763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:00.815686 25763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:00.817070 25763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:00.817145 25763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:00.817158 25763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:00.817170 25763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:00.817181 25763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:00.819085 25763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40397,127.2.251.61:37633,127.2.251.60:41019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:00.819379 25763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:00.819639 25763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:00.825249 25776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:00.825286 25771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:00.825394 25772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:00.825867 25763 server_base.cc:1034] running on GCE node
I20260811 02:40:00.826159 25763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:00.826494 25763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:00.827710 25763 hybrid_clock.cc:648] HybridClock initialized: now 1786416000827641 us; error 78 us; skew 500 ppm
Aug 11 02:40:00 dist-test-slave-cvwc krb5kdc[25741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:00.831022 25763 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:00.831707 25763 webserver.cc:460] Webserver started at http://127.2.251.62:37987/ using document root <none> and password file <none>
I20260811 02:40:00.831990 25763 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:00.832046 25763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:00.832238 25763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:00.833423 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "bf49ac8b683749c28f347225ce527f34"
format_stamp: "Formatted at 2026-08-11 02:40:00 on dist-test-slave-cvwc"
server_key: "e9217f1f1b07435f5cbe84119b5f296e"
server_key_iv: "88903dd07ce1eb91ec98b70ff8e60453"
server_key_version: "encryptionkey@0"
I20260811 02:40:00.833882 25763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "bf49ac8b683749c28f347225ce527f34"
format_stamp: "Formatted at 2026-08-11 02:40:00 on dist-test-slave-cvwc"
server_key: "e9217f1f1b07435f5cbe84119b5f296e"
server_key_iv: "88903dd07ce1eb91ec98b70ff8e60453"
server_key_version: "encryptionkey@0"
I20260811 02:40:00.835598 25763 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260811 02:40:00.836805 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:00.837124 25763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:40:00.837311 25763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "bf49ac8b683749c28f347225ce527f34"
format_stamp: "Formatted at 2026-08-11 02:40:00 on dist-test-slave-cvwc"
server_key: "e9217f1f1b07435f5cbe84119b5f296e"
server_key_iv: "88903dd07ce1eb91ec98b70ff8e60453"
server_key_version: "encryptionkey@0"
I20260811 02:40:00.837411 25763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:00.867991 25763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:00.871166 25763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:00.871361 25763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:00.876396 25763 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40397
I20260811 02:40:00.876528 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40397 every 8 connection(s)
I20260811 02:40:00.876830 25763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:00.877835 25887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:00.879575 25887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:00.883834 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 25763
I20260811 02:40:00.883947 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:40:00.884145 3052 external_mini_cluster.cc:1428] Setting key c30b5535312d69757694ae3bb1750344
Aug 11 02:40:00 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:00.890964 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:00.880427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.2.251.62:40397)
0811 02:40:00.880569 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:00.880577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:00.880656 (+ 79us) server_negotiation.cc:407] Connection header received
0811 02:40:00.881224 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:00.881227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:00.881249 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:00.881340 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:00.882023 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:00.882857 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:00.884159 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:00.884569 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:00.884693 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:00.887582 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:00.887591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:00.887592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:00.887600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:00.889151 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:00.889679 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:00.889682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:00.889682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:00.889811 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:00.890081 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:00.890082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:00.890082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:00.890343 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:00.890406 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:00.890639 (+ 233us) server_negotiation.cc:299] Negotiation successful
0811 02:40:00.890774 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260811 02:40:00.890981 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:00.880350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40397 (local address 127.0.0.1:37236)
0811 02:40:00.880535 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:00.880575 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:00.881046 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:00.881413 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:00.881416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:00.881691 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:00.881916 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:00.881921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:00.883048 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:00.883051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:00.884028 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:00.884034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:00.884664 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:00.884666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:00.884748 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:00.885530 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:00.885543 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:00.887467 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:00.889271 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:00.889281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:00.889285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:00.889574 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:00.889873 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:00.889873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:00.889874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:00.889998 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:00.890417 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:00.890424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:00.890512 (+ 88us) client_negotiation.cc:769] Sending connection context
0811 02:40:00.890621 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 02:40:00.890863 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":66,"thread_start_us":74,"threads_started":1}
I20260811 02:40:00.891809 25887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:00.892200 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:00.892057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49440)
0811 02:40:00.892110 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:00.892170 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 02:40:00.892396 25800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
I20260811 02:40:00.892854 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:00.892750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49454)
0811 02:40:00.892797 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:00.892829 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
WARNING: no policy specified for kudu/127.2.251.61@KRBTEST.COM; defaulting to no policy
W20260811 02:40:00.892959 25887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:37633: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab.
Entry for principal kudu/127.2.251.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab.
I20260811 02:40:00.917373 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:37633
--webserver_interface=127.2.251.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab
--principal=kudu/127.2.251.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40397,127.2.251.61:37633,127.2.251.60:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:00.942148 25887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } attempt: 1
I20260811 02:40:00.942765 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:00.942582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49470)
0811 02:40:00.942647 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:00.942700 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 02:40:00.943153 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:00.943062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49478)
0811 02:40:00.943100 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:00.943131 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 02:40:00.943255 25887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:37633: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
W20260811 02:40:01.006704 25904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:01.006954 25904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:01.006990 25904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:01.008210 25904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:01.008260 25904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:01.008279 25904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:01.008296 25904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:01.008311 25904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:01.010007 25904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40397,127.2.251.61:37633,127.2.251.60:41019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab
--principal=kudu/127.2.251.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:37633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.251.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:01.010279 25904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:01.010502 25904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:40:01.011431 25887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } attempt: 2
I20260811 02:40:01.011938 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.011788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49482)
0811 02:40:01.011850 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.011902 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 02:40:01.012295 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.012204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49492)
0811 02:40:01.012247 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.012273 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260811 02:40:01.012398 25887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:37633: Network error: Client connection negotiation failed: client connection to 127.2.251.61:37633: connect: Connection refused (error 111)
W20260811 02:40:01.016048 25914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:01.016063 25916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:01.016065 25911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:01.016621 25904 server_base.cc:1034] running on GCE node
I20260811 02:40:01.016888 25904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:01.017174 25904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:01.018375 25904 hybrid_clock.cc:648] HybridClock initialized: now 1786416001018358 us; error 28 us; skew 500 ppm
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:01.021235 25904 init.cc:377] Logged in from keytab as kudu/127.2.251.61@KRBTEST.COM (short username kudu)
I20260811 02:40:01.021705 25904 webserver.cc:460] Webserver started at http://127.2.251.61:39087/ using document root <none> and password file <none>
I20260811 02:40:01.021989 25904 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:01.022037 25904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:01.022149 25904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:01.023370 25904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data/instance:
uuid: "66aeb759265a47a68a64865f9450c0a2"
format_stamp: "Formatted at 2026-08-11 02:40:01 on dist-test-slave-cvwc"
server_key: "5f4060f2f95315565c1e5afc7acbbc3e"
server_key_iv: "1f9958fcc80bcf5940b79e011e9769e2"
server_key_version: "encryptionkey@0"
I20260811 02:40:01.023761 25904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal/instance:
uuid: "66aeb759265a47a68a64865f9450c0a2"
format_stamp: "Formatted at 2026-08-11 02:40:01 on dist-test-slave-cvwc"
server_key: "5f4060f2f95315565c1e5afc7acbbc3e"
server_key_iv: "1f9958fcc80bcf5940b79e011e9769e2"
server_key_version: "encryptionkey@0"
I20260811 02:40:01.026163 25904 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 02:40:01.027197 25925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:01.027833 25904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:40:01.027935 25904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal
uuid: "66aeb759265a47a68a64865f9450c0a2"
format_stamp: "Formatted at 2026-08-11 02:40:01 on dist-test-slave-cvwc"
server_key: "5f4060f2f95315565c1e5afc7acbbc3e"
server_key_iv: "1f9958fcc80bcf5940b79e011e9769e2"
server_key_version: "encryptionkey@0"
I20260811 02:40:01.028028 25904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:01.039743 25904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:01.042465 25904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:01.042627 25904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:01.047107 25904 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.61:37633
I20260811 02:40:01.047152 26035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.61:37633 every 8 connection(s)
I20260811 02:40:01.047525 25904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
I20260811 02:40:01.048457 26037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:01.050295 26037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:01.053977 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 25904
I20260811 02:40:01.054091 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-1/wal/instance
I20260811 02:40:01.054275 3052 external_mini_cluster.cc:1428] Setting key 756a4ad8d3793f7c763470d650e19614
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:01.061671 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.051144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.2.251.62:40397)
0811 02:40:01.051249 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.051252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.051441 (+ 189us) server_negotiation.cc:407] Connection header received
0811 02:40:01.051970 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.051971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.051988 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.052042 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.052836 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.053379 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.054579 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.054800 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.054868 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.057392 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.057395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.057396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.057409 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.059658 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.060243 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.060245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.060245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.060328 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.060693 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.060697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.060697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.060898 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.060997 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.061470 (+ 473us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.061590 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 02:40:01.061889 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.051072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40397 (local address 127.0.0.1:37250)
0811 02:40:01.051305 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.051349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.051867 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.052136 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.052139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.052423 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.052702 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.052716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.053488 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.053489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.054495 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.054498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.054882 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.054883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.055020 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.055644 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.055654 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.057179 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.059767 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.059781 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.059785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.060146 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.060397 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.060398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.060399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.060557 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.061005 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.061012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.061244 (+ 232us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.061398 (+ 154us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.061593 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260811 02:40:01.062546 26037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.251.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.60@KRBTEST.COM" created.
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:40:01.071962 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.062865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49508)
0811 02:40:01.062915 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.062926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.062970 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.063334 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.063335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.063485 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.063540 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.063542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.064413 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.064414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.064901 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.064902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.065703 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.065705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.065821 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.066381 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.066386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.068062 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.070654 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.070655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.070656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.070840 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.071155 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.071156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.071156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.071223 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.071660 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.071661 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.071711 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.071776 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.071872 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260811 02:40:01.072086 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.062958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.2.251.61:37633)
0811 02:40:01.063177 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.063182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.063201 (+ 19us) server_negotiation.cc:407] Connection header received
0811 02:40:01.063221 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.063223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.063235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.063318 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.063612 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.064326 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.064971 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.065549 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.065683 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.068207 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.068220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.068220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.068229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.070538 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.070965 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.070969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.070970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.071050 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.071316 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.071318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.071318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.071564 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.071666 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.071830 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.071946 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260811 02:40:01.073338 26037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:01.073753 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.073614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44772)
0811 02:40:01.073658 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.073707 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 02:40:01.073905 25928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
I20260811 02:40:01.074473 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.074311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44774)
0811 02:40:01.074408 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.074444 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260811 02:40:01.074584 26037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41019: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab.
Entry for principal kudu/127.2.251.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab.
I20260811 02:40:01.084966 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:41019
--webserver_interface=127.2.251.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab
--principal=kudu/127.2.251.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40397,127.2.251.61:37633,127.2.251.60:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:01.103631 25887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } attempt: 3
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:40:01.112156 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.104174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.2.251.61:37633)
0811 02:40:01.104237 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.104238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.104253 (+ 15us) server_negotiation.cc:407] Connection header received
0811 02:40:01.104323 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.104323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.104337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.104394 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.104849 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.105385 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.106428 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.106650 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.106721 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.108895 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.108897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.108898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.108906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.110572 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.110965 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.111013 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.111014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.111103 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.111368 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.111369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.111369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.111523 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.111602 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.111953 (+ 351us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.112052 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:40:01.112221 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.104081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49520)
0811 02:40:01.104158 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.104174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.104248 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.104414 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.104415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.104627 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.104713 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.104716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.105479 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.105481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.106316 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.106320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.106736 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.106737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.106818 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.107321 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.107326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.108791 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.110691 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.110693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.110693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.110889 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.111211 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.111212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.111212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.111287 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.111695 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.111698 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.111792 (+ 94us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.111908 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.112115 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 02:40:01.112982 25887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:01.113418 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.113276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44780)
0811 02:40:01.113331 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.113376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 02:40:01.114008 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.113912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44796)
0811 02:40:01.113956 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.113987 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:40:01.114157 25887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41019: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
I20260811 02:40:01.123824 26037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } attempt: 1
I20260811 02:40:01.124311 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.124195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44808)
0811 02:40:01.124249 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.124282 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 02:40:01.124737 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.124655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44816)
0811 02:40:01.124692 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.124718 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260811 02:40:01.124836 26037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41019: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
I20260811 02:40:01.145394 25887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } attempt: 1
I20260811 02:40:01.145982 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.145811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44820)
0811 02:40:01.145883 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.145941 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 02:40:01.146329 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.146241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44836)
0811 02:40:01.146282 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.146309 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:40:01.146421 25887 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41019: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
W20260811 02:40:01.174517 26047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:01.174767 26047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:01.174793 26047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:01.176025 26047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:01.176080 26047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:01.176095 26047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:01.176106 26047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:01.176117 26047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:01.177805 26047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40397,127.2.251.61:37633,127.2.251.60:41019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab
--principal=kudu/127.2.251.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.251.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:01.178069 26047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:01.178324 26047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:01.183490 26068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:01.183492 26060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:01.183511 26047 server_base.cc:1034] running on GCE node
W20260811 02:40:01.183492 26063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:01.183984 26047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:01.184285 26047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:01.185436 26047 hybrid_clock.cc:648] HybridClock initialized: now 1786416001185420 us; error 34 us; skew 500 ppm
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:01.188944 26047 init.cc:377] Logged in from keytab as kudu/127.2.251.60@KRBTEST.COM (short username kudu)
I20260811 02:40:01.189443 26047 webserver.cc:460] Webserver started at http://127.2.251.60:36743/ using document root <none> and password file <none>
I20260811 02:40:01.189675 26047 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:01.189762 26047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:01.189890 26047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:01.191056 26047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data/instance:
uuid: "b41b383af68f4017a2ad79684ae37696"
format_stamp: "Formatted at 2026-08-11 02:40:01 on dist-test-slave-cvwc"
server_key: "ae4d05f5def906f6ed9f14ca0c7e295f"
server_key_iv: "a9b9d9bcad5748e80fad69967c0c2c23"
server_key_version: "encryptionkey@0"
I20260811 02:40:01.191411 26047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal/instance:
uuid: "b41b383af68f4017a2ad79684ae37696"
format_stamp: "Formatted at 2026-08-11 02:40:01 on dist-test-slave-cvwc"
server_key: "ae4d05f5def906f6ed9f14ca0c7e295f"
server_key_iv: "a9b9d9bcad5748e80fad69967c0c2c23"
server_key_version: "encryptionkey@0"
I20260811 02:40:01.192800 26047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:40:01.193001 26037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } attempt: 2
I20260811 02:40:01.193439 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.193311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44848)
0811 02:40:01.193374 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.193411 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 02:40:01.193676 26088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:01.193778 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.193673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44862)
0811 02:40:01.193709 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.193754 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 02:40:01.193897 26037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41019: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41019: connect: Connection refused (error 111)
I20260811 02:40:01.193935 26047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:01.194016 26047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal
uuid: "b41b383af68f4017a2ad79684ae37696"
format_stamp: "Formatted at 2026-08-11 02:40:01 on dist-test-slave-cvwc"
server_key: "ae4d05f5def906f6ed9f14ca0c7e295f"
server_key_iv: "a9b9d9bcad5748e80fad69967c0c2c23"
server_key_version: "encryptionkey@0"
I20260811 02:40:01.194073 26047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:01.206874 26047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:01.209148 26047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:01.209276 26047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:01.213591 26047 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.60:41019
I20260811 02:40:01.213644 26183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.60:41019 every 8 connection(s)
I20260811 02:40:01.214149 26047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
I20260811 02:40:01.215144 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:01.216977 26184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:01.221558 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 26047
I20260811 02:40:01.221557 25887 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } attempt: 2
I20260811 02:40:01.221676 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786415831315317-3052-0/minicluster-data/master-2/wal/instance
I20260811 02:40:01.221922 3052 external_mini_cluster.cc:1428] Setting key 84672fdff4d32cdcc7b53ee026540375
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:40:01.227056 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.217831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.2.251.62:40397)
0811 02:40:01.217915 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.217918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.218111 (+ 193us) server_negotiation.cc:407] Connection header received
0811 02:40:01.218791 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.218792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.218807 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.218860 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.219522 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.220012 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.221004 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.221206 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.221264 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.223885 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.223886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.223887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.223894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.225389 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.225875 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.225877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.225877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.225939 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.226219 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.226220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.226220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.226363 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.226442 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.226862 (+ 420us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.226951 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 02:40:01.227234 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.217708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40397 (local address 127.0.0.1:37258)
0811 02:40:01.217989 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.218027 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.218659 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.218936 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.218939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.219216 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.219431 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.219436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.220097 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.220098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.220922 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.220924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.221280 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.221281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.221402 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.221980 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.221991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.223806 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.225480 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.225489 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.225491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.225809 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.225994 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.225995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.225996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.226141 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.226466 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.226474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.226654 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.226799 (+ 145us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.226985 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260811 02:40:01.227885 26184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:40:01.230242 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.221874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44878)
0811 02:40:01.222076 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.222087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.222133 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.222489 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.222490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.222639 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.222681 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.222683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.223649 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.223650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.224433 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.224435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.225042 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.225043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.225098 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.225441 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.225446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.226815 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.228772 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.228773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.228774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.228994 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.229479 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.229480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.229481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.229554 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.229906 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.229907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.229958 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.230034 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.230135 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260811 02:40:01.230530 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.221954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.2.251.60:41019)
0811 02:40:01.222159 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.222164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.222182 (+ 18us) server_negotiation.cc:407] Connection header received
0811 02:40:01.222206 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.222208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.222355 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.222464 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.222756 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.223562 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.224566 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.224963 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.225027 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.226900 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.226906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.226906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.226913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.228697 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.229307 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.229311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.229311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.229399 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.229615 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.229616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.229616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.229829 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.229913 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.230094 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.230178 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:40:01.232556 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.222242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.2.251.60:41019)
0811 02:40:01.222419 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.222421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.222748 (+ 327us) server_negotiation.cc:407] Connection header received
0811 02:40:01.222767 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.222767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.222778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.222850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.223702 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.224443 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.225654 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.225974 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.226038 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.228827 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.228828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.228828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.228834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.231208 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.231594 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.231595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.231595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.231674 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.231869 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.231870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.231871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.231994 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.232192 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.232254 (+ 62us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.232339 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260811 02:40:01.233084 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: Bootstrap starting.
I20260811 02:40:01.233871 25887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:01.234237 25887 log.cc:824] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:01.235106 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: No bootstrap required, opened a new log
I20260811 02:40:01.236238 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.228488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:37633 (local address 127.0.0.1:49534)
0811 02:40:01.228537 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.228555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.228659 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.228848 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.228849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.228990 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.229039 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.229041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.229688 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.229689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.230400 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.230404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.230825 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.230826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.230899 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.231298 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.231303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.232757 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.234862 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.234863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.234864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.235256 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.235538 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.235539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.235539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.235626 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.235909 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.235910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.235960 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.236030 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.236135 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 02:40:01.236258 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.228570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.2.251.61:37633)
0811 02:40:01.228627 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.228629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.228637 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:40:01.228738 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.228739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.228750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.228780 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.229111 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.229634 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.230479 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.230739 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.230799 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.232830 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.232831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.232832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.232839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.234763 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.235366 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.235368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.235368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.235441 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.235706 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.235707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.235707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.235846 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.235918 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.236066 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.236159 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:40:01.237150 25887 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.237298 25887 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:01.237337 25887 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf49ac8b683749c28f347225ce527f34, State: Initialized, Role: FOLLOWER
I20260811 02:40:01.237334 26184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:40:01.237643 25887 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.238090 25887 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:01.238044 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.238164 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 02:40:01.240432 26220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:40:01.240509 26220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:40:01.241698 25763 master_runner.cc:186] Error getting master registration for 127.2.251.61:37633: OK, instance_id { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" instance_seqno: 1786416001045884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 02:40:01.245308 25763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:40:01.246810 26190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.237582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44910)
0811 02:40:01.237619 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.237630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.237668 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.237906 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.237907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.238054 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.238096 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.238098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.239238 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.239240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.239954 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.239958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.240555 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.240555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.240609 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.241266 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.241272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.243238 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.245675 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.245675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.245676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.245939 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.246129 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.246130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.246130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.246188 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.246475 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.246477 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.246519 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.246580 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.246696 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260811 02:40:01.246866 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.237640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.2.251.60:41019)
0811 02:40:01.237677 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.237679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.237687 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:40:01.237708 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.237709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.237754 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.237802 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.238343 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.239131 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.240105 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.240395 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.240478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.243368 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.243369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.243370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.243378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.245571 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.246016 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.246017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.246017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.246073 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.246238 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.246239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.246239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.246412 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.246475 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.246638 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.246750 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":20}
I20260811 02:40:01.247912 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.234754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.2.251.62:40397)
0811 02:40:01.234795 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.234797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.234838 (+ 41us) server_negotiation.cc:407] Connection header received
0811 02:40:01.234934 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.234935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.234945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.235005 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.235838 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.236484 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.237633 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.237951 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.237996 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.243911 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.243913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.243913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.243921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.246408 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.246860 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.246861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.246861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.246923 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.247404 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.247405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.247405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.247538 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.247665 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.247770 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.247848 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:40:01.248553 26184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: Bootstrap starting.
I20260811 02:40:01.249385 26184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:01.249806 26184 log.cc:824] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:01.250638 26184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: No bootstrap required, opened a new log
I20260811 02:40:01.252682 26184 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.252888 26184 raft_consensus.cc:374] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:01.252928 26184 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41b383af68f4017a2ad79684ae37696, State: Initialized, Role: FOLLOWER
I20260811 02:40:01.253105 26184 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.253507 26184 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:01.253479 26223 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:40:01.254630 26223 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 02:40:01.255986 26246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:40:01.256057 26246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:40:01.260180 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.248784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.2.251.61:37633)
0811 02:40:01.248841 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.248843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.248853 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:40:01.248882 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.248883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.248893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.248946 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.249543 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.250340 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.251392 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.251678 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.251753 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.254965 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.254968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.254969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.254983 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.258207 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.258936 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.258937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.258938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.259013 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.259214 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.259215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.259216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.259368 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.259539 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.259996 (+ 457us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.260080 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:40:01.257985 26047 master_runner.cc:186] Error getting master registration for 127.2.251.61:37633: OK, instance_id { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" instance_seqno: 1786416001045884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 02:40:01.260288 26047 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:40:01.262271 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:40:01.272535 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.263916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.2.251.62:40397)
0811 02:40:01.264165 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.264167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.264176 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:40:01.264200 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.264201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.264212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.264270 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.265035 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.265816 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.266810 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.267104 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.267144 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.269027 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.269029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.269030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.269037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.270971 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.271634 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.271635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.271636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.271701 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.271934 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.271935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.271935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.272062 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.272132 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.272327 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.272429 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260811 02:40:01.275606 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.264382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.2.251.60:41019)
0811 02:40:01.264419 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.264421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.264848 (+ 427us) server_negotiation.cc:407] Connection header received
0811 02:40:01.264940 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.264941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.264951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.265056 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.265898 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.266641 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.267665 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.267967 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.268005 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.269997 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.269998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.269999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.270006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.272244 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.272994 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.272996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.272996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.273059 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.274898 (+ 1839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.274900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.274900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.275113 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.275192 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.275408 (+ 216us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.275505 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:40:01.279979 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.264580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.2.251.61:37633)
0811 02:40:01.264614 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.264616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.268417 (+ 3801us) server_negotiation.cc:407] Connection header received
0811 02:40:01.268536 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.268538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.268550 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.268578 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.270286 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.271010 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.274857 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.275193 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.275266 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.276711 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.276712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.276713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.276721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.278370 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.278923 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.278924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.278924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.278985 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.279399 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.279400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.279400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.279545 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.279644 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.279792 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.279875 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260811 02:40:01.285044 26037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } attempt: 3
Aug 11 02:40:01 dist-test-slave-cvwc krb5kdc[25741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416001, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:40:01.294283 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.285355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41019 (local address 127.0.0.1:44920)
0811 02:40:01.285426 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:01.285437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:01.285473 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:01.285839 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:01.285840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:01.286035 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:01.286092 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.286094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.287033 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.287034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.287842 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:01.287845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.288315 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.288315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:01.288382 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.288900 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:01.288905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:01.290530 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:01.293145 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.293147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.293148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.293374 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.293574 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:01.293575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:01.293575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:01.293631 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:01.293954 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:01.293955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:01.294005 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 02:40:01.294064 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 02:40:01.294175 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 02:40:01.294750 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:01.285543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.2.251.60:41019)
0811 02:40:01.285639 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:01.285641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:01.285649 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:40:01.285674 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:01.285675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:01.285686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:01.285819 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:01.286170 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.286932 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.287928 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:01.288240 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:01.288312 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:01.290646 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:01.290648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:01.290649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:01.290658 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:01.293040 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.293451 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.293452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.293453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.293511 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:01.293690 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:01.293691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:01.293691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:01.293866 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:01.293926 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:01.294563 (+ 637us) server_negotiation.cc:299] Negotiation successful
0811 02:40:01.294648 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 02:40:01.297024 26037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: Bootstrap starting.
I20260811 02:40:01.298754 26037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:01.299144 26037 log.cc:824] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:01.300104 26037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: No bootstrap required, opened a new log
I20260811 02:40:01.302999 26037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.303167 26037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:01.303203 26037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66aeb759265a47a68a64865f9450c0a2, State: Initialized, Role: FOLLOWER
I20260811 02:40:01.303337 26037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.303709 26037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:01.303658 26265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.303859 26265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 02:40:01.305967 26282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:40:01.306083 26282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:40:01.308792 25904 master_runner.cc:263] No leader master found from master 66aeb759265a47a68a64865f9450c0a2
I20260811 02:40:01.308864 25904 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 02:40:01.372555 25803 tablet.cc:2369] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:40:01.460021 26094 tablet.cc:2369] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:40:01.543497 25937 tablet.cc:2369] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:40:01.565382 26265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:01.565814 26265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.566170 26265 leader_election.cc:288] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf49ac8b683749c28f347225ce527f34 (127.2.251.62:40397), b41b383af68f4017a2ad79684ae37696 (127.2.251.60:41019)
I20260811 02:40:01.566629 26139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66aeb759265a47a68a64865f9450c0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41b383af68f4017a2ad79684ae37696" is_pre_election: true
I20260811 02:40:01.566632 25843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66aeb759265a47a68a64865f9450c0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf49ac8b683749c28f347225ce527f34" is_pre_election: true
I20260811 02:40:01.566820 26139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66aeb759265a47a68a64865f9450c0a2 in term 0.
I20260811 02:40:01.566821 25843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66aeb759265a47a68a64865f9450c0a2 in term 0.
I20260811 02:40:01.567054 25928 leader_election.cc:302] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66aeb759265a47a68a64865f9450c0a2, b41b383af68f4017a2ad79684ae37696; no voters:
I20260811 02:40:01.567229 26265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:40:01.567322 26265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:40:01.567346 26265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:01.568248 26265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.568408 26265 leader_election.cc:288] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [CANDIDATE]: Term 1 election: Requested vote from peers bf49ac8b683749c28f347225ce527f34 (127.2.251.62:40397), b41b383af68f4017a2ad79684ae37696 (127.2.251.60:41019)
I20260811 02:40:01.568671 25843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66aeb759265a47a68a64865f9450c0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf49ac8b683749c28f347225ce527f34"
I20260811 02:40:01.568717 26139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66aeb759265a47a68a64865f9450c0a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b41b383af68f4017a2ad79684ae37696"
I20260811 02:40:01.568776 25843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:01.568814 26139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:01.569589 26139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66aeb759265a47a68a64865f9450c0a2 in term 1.
I20260811 02:40:01.569600 25843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66aeb759265a47a68a64865f9450c0a2 in term 1.
I20260811 02:40:01.569814 25928 leader_election.cc:302] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66aeb759265a47a68a64865f9450c0a2, bf49ac8b683749c28f347225ce527f34; no voters:
I20260811 02:40:01.569913 26265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:01.570055 26265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 66aeb759265a47a68a64865f9450c0a2, State: Running, Role: LEADER
I20260811 02:40:01.570152 26265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } }
I20260811 02:40:01.570631 26285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66aeb759265a47a68a64865f9450c0a2. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.570746 26285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:01.571004 26287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:01.571197 26287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:40:01.572870 26139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [term 1 FOLLOWER]: Refusing update from remote peer 66aeb759265a47a68a64865f9450c0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:01.572875 25843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [term 1 FOLLOWER]: Refusing update from remote peer 66aeb759265a47a68a64865f9450c0a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:01.573215 26265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:01.573341 26285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:01.574703 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66aeb759265a47a68a64865f9450c0a2. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.574812 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:40:01.574709 26223 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66aeb759265a47a68a64865f9450c0a2. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.574954 26223 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:40:01.575920 26285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.576021 26285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:01.576071 26285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.576110 26285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:01.576104 26223 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.576185 26223 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:40:01.576170 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66aeb759265a47a68a64865f9450c0a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf49ac8b683749c28f347225ce527f34" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40397 } } peers { permanent_uuid: "66aeb759265a47a68a64865f9450c0a2" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 37633 } } peers { permanent_uuid: "b41b383af68f4017a2ad79684ae37696" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41019 } } }
I20260811 02:40:01.576237 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:40:01.576363 26287 catalog_manager.cc:1347] Generated new cluster ID: afeab50cfc154b8da9ceea837ea3c4e8
I20260811 02:40:01.576419 26287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:40:01.576381 26289 mvcc.cc:205] Tried to move back new op lower bound from 7317159942440214528 to 7317159942432092160. Current Snapshot: MvccSnapshot[applied={T|T < 7317159942440214528}]
I20260811 02:40:01.576480 26290 mvcc.cc:205] Tried to move back new op lower bound from 7317159942440214528 to 7317159942432092160. Current Snapshot: MvccSnapshot[applied={T|T < 7317159942440214528}]
I20260811 02:40:01.586544 26287 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:40:01.587145 26287 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:40:01.591362 26287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66aeb759265a47a68a64865f9450c0a2: Generated new TSK 0
I20260811 02:40:01.591675 26287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:40:02.241180 26220 catalog_manager.cc:1259] Loaded cluster ID: afeab50cfc154b8da9ceea837ea3c4e8
I20260811 02:40:02.241307 26220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: loading cluster ID for follower catalog manager: success
I20260811 02:40:02.242439 26220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: acquiring CA information for follower catalog manager: success
I20260811 02:40:02.242882 26220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf49ac8b683749c28f347225ce527f34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:40:02.256587 26246 catalog_manager.cc:1259] Loaded cluster ID: afeab50cfc154b8da9ceea837ea3c4e8
I20260811 02:40:02.256681 26246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: loading cluster ID for follower catalog manager: success
I20260811 02:40:02.257622 26246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: acquiring CA information for follower catalog manager: success
I20260811 02:40:02.258008 26246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b41b383af68f4017a2ad79684ae37696: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:40:02.649847 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 25763
I20260811 02:40:02.670003 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 25904
I20260811 02:40:02.683094 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 26047
2026-08-11T02:40:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3342 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:40:02 dist-test-slave-cvwc krb5kdc[26311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:40:02 dist-test-slave-cvwc krb5kdc[26311](info): set up 2 sockets
Aug 11 02:40:02 dist-test-slave-cvwc krb5kdc[26311](info): commencing operation
krb5kdc: starting...
W20260811 02:40:03.972623 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:40:04 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:40:04.125634 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39503
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:04.227856 26337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:04.228116 26337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:04.228139 26337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:04.229383 26337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:04.229437 26337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:04.229451 26337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:04.229463 26337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:04.229475 26337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:04.231178 26337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:04.231482 26337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:04.231747 26337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:04.238044 26350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:04.238109 26348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:04.238091 26337 server_base.cc:1034] running on GCE node
W20260811 02:40:04.238083 26347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:04.239357 26337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:04.239848 26337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:04.241087 26337 hybrid_clock.cc:648] HybridClock initialized: now 1786416004241042 us; error 69 us; skew 500 ppm
Aug 11 02:40:04 dist-test-slave-cvwc krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:04.244861 26337 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:04.245755 26337 webserver.cc:460] Webserver started at http://127.2.251.62:40955/ using document root <none> and password file <none>
I20260811 02:40:04.246099 26337 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:04.246248 26337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:04.246487 26337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:04.248311 26337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "e3c46410bd784f1aa9da5316dce3a541"
format_stamp: "Formatted at 2026-08-11 02:40:04 on dist-test-slave-cvwc"
server_key: "64064ad4c9504de50435f3ef560ee975"
server_key_iv: "f4e766938368e7943ecefb081a89606b"
server_key_version: "encryptionkey@0"
I20260811 02:40:04.248713 26337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "e3c46410bd784f1aa9da5316dce3a541"
format_stamp: "Formatted at 2026-08-11 02:40:04 on dist-test-slave-cvwc"
server_key: "64064ad4c9504de50435f3ef560ee975"
server_key_iv: "f4e766938368e7943ecefb081a89606b"
server_key_version: "encryptionkey@0"
I20260811 02:40:04.250566 26337 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 02:40:04.251577 26362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:04.251782 26337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:04.251909 26337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "e3c46410bd784f1aa9da5316dce3a541"
format_stamp: "Formatted at 2026-08-11 02:40:04 on dist-test-slave-cvwc"
server_key: "64064ad4c9504de50435f3ef560ee975"
server_key_iv: "f4e766938368e7943ecefb081a89606b"
server_key_version: "encryptionkey@0"
I20260811 02:40:04.252130 26337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:04.272609 26337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:04.275661 26337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:04.275836 26337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:04.281883 26337 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39503
I20260811 02:40:04.282002 26450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39503 every 8 connection(s)
I20260811 02:40:04.282501 26337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:04.283263 26455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:04.283900 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 26337
I20260811 02:40:04.283998 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:40:04.284174 3052 external_mini_cluster.cc:1428] Setting key 4e2c60fee37a67cf2e1fd9c57c24c35f
I20260811 02:40:04.286331 26455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541: Bootstrap starting.
I20260811 02:40:04.287024 26455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:04.287305 26455 log.cc:824] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:04.288143 26455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541: No bootstrap required, opened a new log
I20260811 02:40:04.290745 26455 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c46410bd784f1aa9da5316dce3a541" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39503 } }
I20260811 02:40:04.290944 26455 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:04.290969 26455 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c46410bd784f1aa9da5316dce3a541, State: Initialized, Role: FOLLOWER
I20260811 02:40:04.291095 26455 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c46410bd784f1aa9da5316dce3a541" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39503 } }
I20260811 02:40:04.291168 26455 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:40:04.291196 26455 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:40:04.291246 26455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 02:40:04 dist-test-slave-cvwc krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:04.292122 26455 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c46410bd784f1aa9da5316dce3a541" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39503 } }
I20260811 02:40:04.292260 26455 leader_election.cc:302] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c46410bd784f1aa9da5316dce3a541; no voters:
I20260811 02:40:04.292848 26455 leader_election.cc:288] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:40:04.292909 26464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:04.293071 26464 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [term 1 LEADER]: Becoming Leader. State: Replica: e3c46410bd784f1aa9da5316dce3a541, State: Running, Role: LEADER
I20260811 02:40:04.293246 26455 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:04.293268 26464 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c46410bd784f1aa9da5316dce3a541" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39503 } }
I20260811 02:40:04.293843 26467 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c46410bd784f1aa9da5316dce3a541. Latest consensus state: current_term: 1 leader_uuid: "e3c46410bd784f1aa9da5316dce3a541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c46410bd784f1aa9da5316dce3a541" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39503 } } }
I20260811 02:40:04.294050 26467 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:04.294488 26476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:04.295187 26476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:40:04.295240 26465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c46410bd784f1aa9da5316dce3a541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c46410bd784f1aa9da5316dce3a541" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39503 } } }
I20260811 02:40:04.296373 26465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:04.297014 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:04.284735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.2.251.62:39503)
0811 02:40:04.284898 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:04.284906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:04.284933 (+ 27us) server_negotiation.cc:407] Connection header received
0811 02:40:04.285371 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:04.285378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:04.285427 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:04.285544 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:04.286438 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:04.287444 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:04.288458 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:04.288890 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:04.288975 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:04.292001 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:04.292006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:04.292007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:04.292024 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:04.294847 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:04.295315 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:04.295319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:04.295319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:04.295453 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:04.295702 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:04.295703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:04.295703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:04.296003 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:04.296160 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:04.296616 (+ 456us) server_negotiation.cc:299] Negotiation successful
0811 02:40:04.296787 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260811 02:40:04.297510 26476 catalog_manager.cc:1347] Generated new cluster ID: e6ac166f47ef4863a2a4e2c2463e6f61
I20260811 02:40:04.297590 26476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:40:04.298382 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:40:04.315025 26476 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:40:04.315685 26476 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:40:04.322644 26476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c46410bd784f1aa9da5316dce3a541: Generated new TSK 0
I20260811 02:40:04.322866 26476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:40:04.352818 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 26337
2026-08-11T02:40:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1689 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:40:04 dist-test-slave-cvwc krb5kdc[26493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:40:04 dist-test-slave-cvwc krb5kdc[26493](info): set up 2 sockets
Aug 11 02:40:04 dist-test-slave-cvwc krb5kdc[26493](info): commencing operation
krb5kdc: starting...
W20260811 02:40:05.635499 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:40:05 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:40:05.768895 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40427
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:05.859416 26519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:05.859670 26519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:05.859695 26519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:05.860898 26519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:05.860949 26519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:05.860963 26519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:05.860975 26519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:05.860985 26519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:05.862628 26519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:05.862912 26519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:05.863149 26519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:05.868584 26529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:05.868584 26531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:05.868813 26539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:05.868902 26519 server_base.cc:1034] running on GCE node
I20260811 02:40:05.869172 26519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:05.869499 26519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:05.870680 26519 hybrid_clock.cc:648] HybridClock initialized: now 1786416005870647 us; error 73 us; skew 500 ppm
Aug 11 02:40:05 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416005, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:05.873392 26519 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:05.873921 26519 webserver.cc:460] Webserver started at http://127.2.251.62:46311/ using document root <none> and password file <none>
I20260811 02:40:05.874143 26519 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:05.874193 26519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:05.874333 26519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:05.875497 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "8911d4bce42643e8a9a01d32602d5e46"
format_stamp: "Formatted at 2026-08-11 02:40:05 on dist-test-slave-cvwc"
server_key: "19ad009a7ba1c7c9c6398bd11b2d81a1"
server_key_iv: "eff59ca8a968f8bb95194edc9b4b8bc1"
server_key_version: "encryptionkey@0"
I20260811 02:40:05.875860 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "8911d4bce42643e8a9a01d32602d5e46"
format_stamp: "Formatted at 2026-08-11 02:40:05 on dist-test-slave-cvwc"
server_key: "19ad009a7ba1c7c9c6398bd11b2d81a1"
server_key_iv: "eff59ca8a968f8bb95194edc9b4b8bc1"
server_key_version: "encryptionkey@0"
I20260811 02:40:05.877261 26519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:40:05.878392 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:05.878656 26519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:05.878767 26519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "8911d4bce42643e8a9a01d32602d5e46"
format_stamp: "Formatted at 2026-08-11 02:40:05 on dist-test-slave-cvwc"
server_key: "19ad009a7ba1c7c9c6398bd11b2d81a1"
server_key_iv: "eff59ca8a968f8bb95194edc9b4b8bc1"
server_key_version: "encryptionkey@0"
I20260811 02:40:05.878840 26519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:05.936335 26519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:05.938606 26519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:05.938768 26519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:05.942773 26519 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40427
I20260811 02:40:05.942819 26632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40427 every 8 connection(s)
I20260811 02:40:05.943153 26519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:05.944367 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:05.946524 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 26519
I20260811 02:40:05.946645 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:40:05.946933 3052 external_mini_cluster.cc:1428] Setting key 33872ab0518bede3ec13a1fb3107ab8b
I20260811 02:40:05.947184 26633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap starting.
I20260811 02:40:05.948024 26633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:05.948369 26633 log.cc:824] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:05.949131 26633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: No bootstrap required, opened a new log
I20260811 02:40:05.950995 26633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:05.951140 26633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:05.951162 26633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8911d4bce42643e8a9a01d32602d5e46, State: Initialized, Role: FOLLOWER
I20260811 02:40:05.951282 26633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:05.951364 26633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:40:05.951407 26633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:40:05.951455 26633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:05.952040 26633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:05.952143 26633 leader_election.cc:302] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8911d4bce42643e8a9a01d32602d5e46; no voters:
I20260811 02:40:05.952337 26633 leader_election.cc:288] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:40:05.952404 26642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:05.952613 26633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:05.952631 26642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 LEADER]: Becoming Leader. State: Replica: 8911d4bce42643e8a9a01d32602d5e46, State: Running, Role: LEADER
I20260811 02:40:05.952749 26642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:05.953133 26642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8911d4bce42643e8a9a01d32602d5e46. Latest consensus state: current_term: 1 leader_uuid: "8911d4bce42643e8a9a01d32602d5e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } } }
I20260811 02:40:05.953210 26642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:05.953372 26643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8911d4bce42643e8a9a01d32602d5e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } } }
I20260811 02:40:05.953431 26643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:05.953485 26659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:05.954097 26659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 11 02:40:05 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:05.956120 26659 catalog_manager.cc:1347] Generated new cluster ID: b50b6e0b021c4b3bacd404d5f4954b2b
I20260811 02:40:05.956171 26659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:40:05.962730 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:05.947566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.2.251.62:40427)
0811 02:40:05.947797 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:05.947807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:05.947846 (+ 39us) server_negotiation.cc:407] Connection header received
0811 02:40:05.948505 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:05.948516 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:05.948585 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:05.948750 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:05.949497 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:05.950442 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:05.951257 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:05.951758 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:05.951869 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:05.954958 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:05.954965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:05.954966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:05.954979 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:05.959894 (+ 4915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:05.960770 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:05.960775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:05.960776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:05.960921 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:05.961152 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:05.961153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:05.961154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:05.961518 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:05.962249 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:05.962341 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 02:40:05.962497 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260811 02:40:05.970268 26659 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:40:05.970850 26659 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:40:05.976258 26659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Generated new TSK 0
I20260811 02:40:05.976550 26659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:40:06.011265 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:40427
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:06.108705 26691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:06.108953 26691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:06.108989 26691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:06.110291 26691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:06.110345 26691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:06.110386 26691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:40:06.112044 26691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:06.112358 26691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:06.112610 26691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:06.119208 26696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.119333 26702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:06.119524 26691 server_base.cc:1034] running on GCE node
W20260811 02:40:06.119230 26697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:06.119908 26691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:06.120189 26691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:06.121402 26691 hybrid_clock.cc:648] HybridClock initialized: now 1786416006121369 us; error 42 us; skew 500 ppm
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:06.124884 26691 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:40:06.125432 26691 webserver.cc:460] Webserver started at http://127.2.251.1:46681/ using document root <none> and password file <none>
I20260811 02:40:06.125674 26691 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:06.125752 26691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:06.125885 26691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:06.126997 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "00fa813265054041b4c90a3146983f81"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "93be7f7fec47cdd92095650c6067e024"
server_key_iv: "48842e4f1036266be755af9edbd76120"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.127326 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "00fa813265054041b4c90a3146983f81"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "93be7f7fec47cdd92095650c6067e024"
server_key_iv: "48842e4f1036266be755af9edbd76120"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.128953 26691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:40:06.129828 26715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.130142 26691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:06.130247 26691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "00fa813265054041b4c90a3146983f81"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "93be7f7fec47cdd92095650c6067e024"
server_key_iv: "48842e4f1036266be755af9edbd76120"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.130328 26691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:06.143904 26691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:06.147176 26691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:06.147364 26691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:06.147668 26691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:06.148059 26691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:06.148100 26691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.148135 26691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:06.148164 26691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.154906 26691 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:42551
I20260811 02:40:06.154986 26904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:42551 every 8 connection(s)
I20260811 02:40:06.155318 26691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:40:06.159034 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 26691
I20260811 02:40:06.159140 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:40:06.159329 3052 external_mini_cluster.cc:1428] Setting key b9945555c66de7f30abf4f264a4dca0e
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:06.166844 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.156177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:33895 (local address 127.2.251.62:40427)
0811 02:40:06.156272 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.156274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.156458 (+ 184us) server_negotiation.cc:407] Connection header received
0811 02:40:06.156953 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.156954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.156966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.157025 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:06.158078 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.158614 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.159975 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.160263 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.160328 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.163198 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:06.163200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:06.163200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:06.163210 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:06.164974 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.165662 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.165664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.165664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.165755 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.166030 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.166030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.166031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.166189 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:06.166272 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.166623 (+ 351us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.166734 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 02:40:06.166976 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.156109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:33895)
0811 02:40:06.156292 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.156336 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.156849 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.157094 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.157098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.157733 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:06.157957 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.157963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.158726 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.158727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.159884 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.159888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.160352 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.160353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.160501 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.161097 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:06.161108 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:06.163015 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:06.165123 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:06.165135 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:06.165138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:06.165569 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:06.165840 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:06.165841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:06.165841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:06.165964 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:06.166304 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:06.166311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:06.166469 (+ 158us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.166585 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.166754 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260811 02:40:06.167636 26907 heartbeater.cc:347] Connected to a master server at 127.2.251.62:40427
I20260811 02:40:06.167759 26907 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:06.167968 26907 heartbeater.cc:511] Master 127.2.251.62:40427 requested a full tablet report, sending...
I20260811 02:40:06.168684 26580 ts_manager.cc:194] Registered new tserver with Master: 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551)
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:40:06.169852 26580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:33895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:40:06.193940 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:40427
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:06.288237 26912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:06.288489 26912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:06.288516 26912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:06.289745 26912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:06.289793 26912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:06.289825 26912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:40:06.291419 26912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:06.291651 26912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:06.291857 26912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:06.297284 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.297484 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.297289 26920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:06.297484 26912 server_base.cc:1034] running on GCE node
I20260811 02:40:06.297863 26912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:06.298123 26912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:06.299381 26912 hybrid_clock.cc:648] HybridClock initialized: now 1786416006299360 us; error 32 us; skew 500 ppm
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:06.302325 26912 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:40:06.302875 26912 webserver.cc:460] Webserver started at http://127.2.251.2:36507/ using document root <none> and password file <none>
I20260811 02:40:06.303103 26912 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:06.303153 26912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:06.303292 26912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:06.304437 26912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "dbf82d620d17486a9757a7a97b5f767f"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "7cf40692de6bba9eed9d8eafd01fb8b9"
server_key_iv: "5fcb165475e93f426380680aa3a128aa"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.304833 26912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "dbf82d620d17486a9757a7a97b5f767f"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "7cf40692de6bba9eed9d8eafd01fb8b9"
server_key_iv: "5fcb165475e93f426380680aa3a128aa"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.306602 26912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:40:06.307593 26939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.307978 26912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:06.308104 26912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "dbf82d620d17486a9757a7a97b5f767f"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "7cf40692de6bba9eed9d8eafd01fb8b9"
server_key_iv: "5fcb165475e93f426380680aa3a128aa"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.308189 26912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:06.327196 26912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:06.329274 26912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:06.329429 26912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:06.329694 26912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:06.330065 26912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:06.330101 26912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.330130 26912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:06.330153 26912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.336547 26912 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:36753
I20260811 02:40:06.336673 27121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:36753 every 8 connection(s)
I20260811 02:40:06.336972 26912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:40:06.342511 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 26912
I20260811 02:40:06.342646 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:40:06.342835 3052 external_mini_cluster.cc:1428] Setting key 56de2cb8f44190b4c7b7a485fa359293
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:06.348469 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.337858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:54263 (local address 127.2.251.62:40427)
0811 02:40:06.337924 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.337970 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.338150 (+ 180us) server_negotiation.cc:407] Connection header received
0811 02:40:06.338704 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.338705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.338716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.338773 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:06.339801 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.340319 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.341409 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.341643 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.341845 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.344885 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:06.344887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:06.344887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:06.344898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:06.346776 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.347297 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.347299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.347299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.347383 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.347648 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.347648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.347649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.347811 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:06.347888 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.348296 (+ 408us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.348393 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:40:06.348708 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.337761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:54263)
0811 02:40:06.338006 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.338054 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.338593 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.338850 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.338854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.339462 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:06.339671 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.339676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.340439 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.340440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.341306 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.341309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.341934 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.341936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.342152 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.343124 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:06.343138 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:06.344767 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:06.346929 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:06.346939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:06.346941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:06.347210 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:06.347455 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:06.347455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:06.347456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:06.347580 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:06.347912 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:06.347922 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:06.348121 (+ 199us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.348246 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.348454 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260811 02:40:06.349287 27123 heartbeater.cc:347] Connected to a master server at 127.2.251.62:40427
I20260811 02:40:06.349404 27123 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:06.349601 27123 heartbeater.cc:511] Master 127.2.251.62:40427 requested a full tablet report, sending...
I20260811 02:40:06.350093 26580 ts_manager.cc:194] Registered new tserver with Master: dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
I20260811 02:40:06.350492 26580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:54263
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:40:06.376577 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:40427
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:06.465692 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:06.465961 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:06.465987 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:06.467190 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:06.467235 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:06.467262 27133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:40:06.468807 27133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:06.469029 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:06.469223 27133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:06.474728 27151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.474704 27143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.474828 27144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:06.475349 27133 server_base.cc:1034] running on GCE node
I20260811 02:40:06.475591 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:06.475871 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:06.477041 27133 hybrid_clock.cc:648] HybridClock initialized: now 1786416006477025 us; error 35 us; skew 500 ppm
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:06.479821 27133 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:40:06.480273 27133 webserver.cc:460] Webserver started at http://127.2.251.3:41213/ using document root <none> and password file <none>
I20260811 02:40:06.480513 27133 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:06.480561 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:06.480681 27133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:06.481842 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "2c988bdc99b14194b204c06b24c77b1a"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "db3570ad039e1f318d565f9f78864c94"
server_key_iv: "3bf10bd0cc48220b98981533e94be448"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.482174 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "2c988bdc99b14194b204c06b24c77b1a"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "db3570ad039e1f318d565f9f78864c94"
server_key_iv: "3bf10bd0cc48220b98981533e94be448"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.483770 27133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:06.484709 27162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.485167 27133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:06.485338 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "2c988bdc99b14194b204c06b24c77b1a"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "db3570ad039e1f318d565f9f78864c94"
server_key_iv: "3bf10bd0cc48220b98981533e94be448"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.485446 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:06.505810 27133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:06.508437 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:06.508577 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:06.508811 27133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:06.509117 27133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:06.509153 27133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.509176 27133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:06.509187 27133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.515798 27133 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:34441
I20260811 02:40:06.515906 27340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:34441 every 8 connection(s)
I20260811 02:40:06.516222 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:06.523393 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 27133
I20260811 02:40:06.523499 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:40:06.523669 3052 external_mini_cluster.cc:1428] Setting key f11f5a8729b4351ba77c75b552ac66be
I20260811 02:40:06.526415 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.516956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:32969)
0811 02:40:06.517123 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.517164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.517685 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.517939 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.517942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.518555 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:06.518762 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.518766 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.519552 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.519553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.520375 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.520377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.520780 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.520781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.520900 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.521481 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:06.521490 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:06.523091 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:06.525002 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:06.525010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:06.525012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:06.525274 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:06.525487 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:06.525487 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:06.525488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:06.525587 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:06.525843 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:06.525847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:06.525979 (+ 132us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.526074 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.526216 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260811 02:40:06.526444 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.517031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:32969 (local address 127.2.251.62:40427)
0811 02:40:06.517114 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.517116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.517247 (+ 131us) server_negotiation.cc:407] Connection header received
0811 02:40:06.517806 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.517808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.517821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.517879 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:06.518866 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.519394 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.520469 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.520672 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.520737 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.523218 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:06.523220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:06.523221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:06.523230 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:06.524935 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.525369 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.525371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.525371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.525434 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.525629 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.525630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.525630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.525784 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:06.525852 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.526208 (+ 356us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.526350 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 02:40:06.527136 27343 heartbeater.cc:347] Connected to a master server at 127.2.251.62:40427
I20260811 02:40:06.527263 27343 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:06.527681 27343 heartbeater.cc:511] Master 127.2.251.62:40427 requested a full tablet report, sending...
I20260811 02:40:06.528282 26580 ts_manager.cc:194] Registered new tserver with Master: 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441)
I20260811 02:40:06.528703 26580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:32969
I20260811 02:40:06.536316 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:40:06.544147 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.538005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.2.251.62:40427)
0811 02:40:06.538092 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.538094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.538139 (+ 45us) server_negotiation.cc:407] Connection header received
0811 02:40:06.538225 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.538226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.538237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.538286 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:06.538931 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.539444 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.540167 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.540371 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.540428 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.541640 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:06.541641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:06.541642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:06.541650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:06.542977 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.543301 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.543302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.543302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.543361 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:06.543576 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:06.543577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:06.543577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:06.543748 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:06.543846 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.543960 (+ 114us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.544046 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 02:40:06.546608 26580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:40:06.548280 26580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:40:06.555605 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.551039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:36753 (local address 127.0.0.1:55810)
0811 02:40:06.551389 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.551399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.551442 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.551889 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.551893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.551897 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:06.552005 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.552008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.553433 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.553434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.554482 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.554486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.555384 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.555385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.555441 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.555443 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.555492 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.555523 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":30,"thread_start_us":108,"threads_started":1}
I20260811 02:40:06.555652 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.551425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.2.251.2:36753)
0811 02:40:06.551617 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.551620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.551630 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:40:06.551679 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.551680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.551696 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.551801 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:06.552090 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.553279 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.554557 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.555294 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.555385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.555450 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.555509 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.555559 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20260811 02:40:06.555795 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.551395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.2.251.3:34441)
0811 02:40:06.551613 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.551619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.551630 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:40:06.551678 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.551680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.551696 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.551791 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:06.552090 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.553280 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.554666 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.555174 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.555241 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.555312 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.555655 (+ 343us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.555707 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260811 02:40:06.555939 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.551155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.2.251.1:42551)
0811 02:40:06.551696 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.551700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.551707 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:40:06.551743 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.551745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.551758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.551827 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:06.552069 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.552962 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.554632 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.555348 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.555407 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.555455 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.555794 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.555845 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":92,"threads_started":1}
I20260811 02:40:06.555728 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.551035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42551 (local address 127.0.0.1:44068)
0811 02:40:06.551236 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.551277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.551438 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.551909 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.551911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.551913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:06.551993 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.551996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.553065 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.553067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.554552 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.554556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.555493 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.555494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.555558 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.555562 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.555611 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.555657 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":75,"thread_start_us":76,"threads_started":1}
I20260811 02:40:06.555612 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.551039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34441 (local address 127.0.0.1:34632)
0811 02:40:06.551545 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.551553 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.551585 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.551889 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.551893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.551895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:06.552005 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.552008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.553401 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.553402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.554580 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.554582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.555265 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.555266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.555349 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.555366 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.555435 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.555499 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":30,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260811 02:40:06.556967 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 490eea2c622f44f1b928104377cf4c7d (DEFAULT_TABLE table=test-table [id=0a5792c138a94c07937efff2930f8cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:06.556970 27016 tablet_service.cc:1467] Processing CreateTablet for tablet 490eea2c622f44f1b928104377cf4c7d (DEFAULT_TABLE table=test-table [id=0a5792c138a94c07937efff2930f8cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:06.557104 26794 tablet_service.cc:1467] Processing CreateTablet for tablet 490eea2c622f44f1b928104377cf4c7d (DEFAULT_TABLE table=test-table [id=0a5792c138a94c07937efff2930f8cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:06.557346 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490eea2c622f44f1b928104377cf4c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:06.557365 27016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490eea2c622f44f1b928104377cf4c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:06.557387 26794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 490eea2c622f44f1b928104377cf4c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:06.560130 27367 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Bootstrap starting.
I20260811 02:40:06.560679 27371 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Bootstrap starting.
I20260811 02:40:06.560948 27367 tablet_bootstrap.cc:654] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:06.561342 27371 tablet_bootstrap.cc:654] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:06.561578 27367 log.cc:824] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:06.561873 27369 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Bootstrap starting.
I20260811 02:40:06.561972 27371 log.cc:824] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:06.562458 27367 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: No bootstrap required, opened a new log
I20260811 02:40:06.562536 27367 ts_tablet_manager.cc:1397] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:40:06.562548 27369 tablet_bootstrap.cc:654] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:06.562759 27371 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: No bootstrap required, opened a new log
I20260811 02:40:06.562827 27371 ts_tablet_manager.cc:1397] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:40:06.563015 27369 log.cc:824] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:06.563661 27369 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: No bootstrap required, opened a new log
I20260811 02:40:06.563721 27369 ts_tablet_manager.cc:1397] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:40:06.564543 27367 raft_consensus.cc:348] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.564744 27367 raft_consensus.cc:374] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:06.564797 27367 raft_consensus.cc:729] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c988bdc99b14194b204c06b24c77b1a, State: Initialized, Role: FOLLOWER
I20260811 02:40:06.564746 27371 raft_consensus.cc:348] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.564929 27371 raft_consensus.cc:374] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:06.564972 27371 raft_consensus.cc:729] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fa813265054041b4c90a3146983f81, State: Initialized, Role: FOLLOWER
I20260811 02:40:06.564950 27367 consensus_queue.cc:260] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.565081 27371 consensus_queue.cc:260] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.565320 27367 ts_tablet_manager.cc:1428] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:40:06.565349 27371 ts_tablet_manager.cc:1428] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:40:06.565408 26907 heartbeater.cc:503] Master 127.2.251.62:40427 was elected leader, sending a full tablet report...
I20260811 02:40:06.565521 27369 raft_consensus.cc:348] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.565670 27369 raft_consensus.cc:374] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:06.565706 27369 raft_consensus.cc:729] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf82d620d17486a9757a7a97b5f767f, State: Initialized, Role: FOLLOWER
I20260811 02:40:06.565703 27343 heartbeater.cc:503] Master 127.2.251.62:40427 was elected leader, sending a full tablet report...
I20260811 02:40:06.565862 27369 consensus_queue.cc:260] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.566139 27369 ts_tablet_manager.cc:1428] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260811 02:40:06.566243 27123 heartbeater.cc:503] Master 127.2.251.62:40427 was elected leader, sending a full tablet report...
W20260811 02:40:06.587585 27124 tablet.cc:2369] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:40:06.656232 26908 tablet.cc:2369] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:40:06.677688 27377 raft_consensus.cc:482] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:06.677896 27377 raft_consensus.cc:504] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.678261 27377 leader_election.cc:288] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
I20260811 02:40:06.681461 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.678545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:56715 (local address 127.2.251.1:42551)
0811 02:40:06.678599 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.678602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.678613 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:40:06.678683 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.678684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.678696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.678751 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:06.678925 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.679715 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.680722 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.681099 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.681161 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.681174 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.681344 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.681385 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:40:06.681460 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.678563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:37455 (local address 127.2.251.2:36753)
0811 02:40:06.678608 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.678610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.678801 (+ 191us) server_negotiation.cc:407] Connection header received
0811 02:40:06.678846 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.678847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.678859 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.678909 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:06.679037 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.679714 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.680707 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.681134 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.681196 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.681207 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.681349 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.681389 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:40:06.681499 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.678483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42551 (local address 127.2.251.3:56715)
0811 02:40:06.678556 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.678568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.678619 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.678779 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.678780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.678783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:06.678857 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.678859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.679835 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.679836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.680638 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.680641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.681199 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.681200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.681279 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.681282 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.681329 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.681376 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":4}
I20260811 02:40:06.681538 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.678496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:36753 (local address 127.2.251.3:37455)
0811 02:40:06.678722 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.678739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:06.678786 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:06.678921 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:06.678922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:06.678923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:06.678984 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.678985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.679850 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.679852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.680613 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:06.680615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.681203 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.681204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:06.681278 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.681281 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:40:06.681327 (+ 46us) client_negotiation.cc:240] Negotiation successful
0811 02:40:06.681372 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3,"thread_start_us":149,"threads_started":1}
I20260811 02:40:06.681945 27051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490eea2c622f44f1b928104377cf4c7d" candidate_uuid: "2c988bdc99b14194b204c06b24c77b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf82d620d17486a9757a7a97b5f767f" is_pre_election: true
I20260811 02:40:06.682016 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490eea2c622f44f1b928104377cf4c7d" candidate_uuid: "2c988bdc99b14194b204c06b24c77b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fa813265054041b4c90a3146983f81" is_pre_election: true
I20260811 02:40:06.682116 27051 raft_consensus.cc:2454] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c988bdc99b14194b204c06b24c77b1a in term 0.
I20260811 02:40:06.682170 26822 raft_consensus.cc:2454] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c988bdc99b14194b204c06b24c77b1a in term 0.
I20260811 02:40:06.682423 27164 leader_election.cc:302] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f; no voters:
I20260811 02:40:06.682613 27377 raft_consensus.cc:2781] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:40:06.682689 27377 raft_consensus.cc:482] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:40:06.682713 27377 raft_consensus.cc:3037] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:06.683509 27377 raft_consensus.cc:504] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.683658 27377 leader_election.cc:288] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [CANDIDATE]: Term 1 election: Requested vote from peers 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
I20260811 02:40:06.683841 27051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490eea2c622f44f1b928104377cf4c7d" candidate_uuid: "2c988bdc99b14194b204c06b24c77b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf82d620d17486a9757a7a97b5f767f"
I20260811 02:40:06.683940 27051 raft_consensus.cc:3037] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:06.684119 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "490eea2c622f44f1b928104377cf4c7d" candidate_uuid: "2c988bdc99b14194b204c06b24c77b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fa813265054041b4c90a3146983f81"
I20260811 02:40:06.684221 26822 raft_consensus.cc:3037] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:06.684700 27051 raft_consensus.cc:2454] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c988bdc99b14194b204c06b24c77b1a in term 1.
I20260811 02:40:06.684898 27164 leader_election.cc:302] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f; no voters:
I20260811 02:40:06.684990 27377 raft_consensus.cc:2781] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:06.685163 27377 raft_consensus.cc:686] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 1 LEADER]: Becoming Leader. State: Replica: 2c988bdc99b14194b204c06b24c77b1a, State: Running, Role: LEADER
I20260811 02:40:06.685184 26822 raft_consensus.cc:2454] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c988bdc99b14194b204c06b24c77b1a in term 1.
I20260811 02:40:06.685331 27377 consensus_queue.cc:237] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:06.686404 26578 catalog_manager.cc:5495] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3). New cstate: current_term: 1 leader_uuid: "2c988bdc99b14194b204c06b24c77b1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } health_report { overall_health: UNKNOWN } } }
I20260811 02:40:06.706763 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.703705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.2.251.3:34441)
0811 02:40:06.703789 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:06.703791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:06.703799 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:40:06.703861 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:06.703862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:06.703892 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:06.703973 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:40:06.704220 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.704892 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.705843 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:06.706060 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:06.706139 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:06.706365 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:40:06.706492 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:40:06.706576 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:06.706649 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 02:40:06.706674 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 02:40:06.708847 27051 raft_consensus.cc:1261] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Refusing update from remote peer 2c988bdc99b14194b204c06b24c77b1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:06.708846 26822 raft_consensus.cc:1261] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Refusing update from remote peer 2c988bdc99b14194b204c06b24c77b1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:06.709270 27377 consensus_queue.cc:1035] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:06.709509 27381 consensus_queue.cc:1035] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:06.713361 27387 mvcc.cc:205] Tried to move back new op lower bound from 7317159963476353024 to 7317159963383947264. Current Snapshot: MvccSnapshot[applied={T|T < 7317159963476353024}]
I20260811 02:40:06.713941 27386 mvcc.cc:205] Tried to move back new op lower bound from 7317159963476353024 to 7317159963383947264. Current Snapshot: MvccSnapshot[applied={T|T < 7317159963476353024}]
I20260811 02:40:06.715443 26580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:06.715533 26580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:06.716300 26580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Sending DeleteTablet for 3 replicas of tablet 490eea2c622f44f1b928104377cf4c7d
I20260811 02:40:06.716681 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 26691
I20260811 02:40:06.716877 27016 tablet_service.cc:1514] Processing DeleteTablet for tablet 490eea2c622f44f1b928104377cf4c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:06 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:55810
I20260811 02:40:06.717175 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 490eea2c622f44f1b928104377cf4c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:06 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:34632
I20260811 02:40:06.717424 27403 tablet_replica.cc:330] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: stopping tablet replica
I20260811 02:40:06.717428 27401 tablet_replica.cc:330] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: stopping tablet replica
I20260811 02:40:06.717526 27403 raft_consensus.cc:2229] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:40:06.717549 27401 raft_consensus.cc:2229] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:40:06.717620 27403 raft_consensus.cc:2258] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:06.717684 27401 raft_consensus.cc:2258] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:06.720458 27403 ts_tablet_manager.cc:1905] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:06.722393 27403 ts_tablet_manager.cc:1918] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:06.722476 27403 log.cc:1196] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/490eea2c622f44f1b928104377cf4c7d
I20260811 02:40:06.722759 27403 ts_tablet_manager.cc:1939] T 490eea2c622f44f1b928104377cf4c7d P 2c988bdc99b14194b204c06b24c77b1a: Deleting consensus metadata
I20260811 02:40:06.724745 26557 catalog_manager.cc:4841] TS 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): tablet 490eea2c622f44f1b928104377cf4c7d (table test-table [id=0a5792c138a94c07937efff2930f8cab]) successfully deleted
I20260811 02:40:06.726548 27401 ts_tablet_manager.cc:1905] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:06.728514 27401 ts_tablet_manager.cc:1918] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:06.728611 27401 log.cc:1196] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/490eea2c622f44f1b928104377cf4c7d
I20260811 02:40:06.728978 27401 ts_tablet_manager.cc:1939] T 490eea2c622f44f1b928104377cf4c7d P dbf82d620d17486a9757a7a97b5f767f: Deleting consensus metadata
I20260811 02:40:06.729619 26554 catalog_manager.cc:4841] TS dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): tablet 490eea2c622f44f1b928104377cf4c7d (table test-table [id=0a5792c138a94c07937efff2930f8cab]) successfully deleted
W20260811 02:40:06.740650 26555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:42551 (error 108)
I20260811 02:40:06.741272 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:06.741131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42551 (local address 127.0.0.1:44080)
0811 02:40:06.741192 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:06.741234 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260811 02:40:06.741358 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 26912
W20260811 02:40:06.749578 26555 catalog_manager.cc:4584] TS 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 490eea2c622f44f1b928104377cf4c7d: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42551: connect: Connection refused (error 111)
I20260811 02:40:06.759776 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 27133
I20260811 02:40:06.777236 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 26519
I20260811 02:40:06.795608 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40427
--webserver_interface=127.2.251.62
--webserver_port=46311
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:06.884796 27405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:06.885033 27405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:06.885066 27405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:06.886328 27405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:06.886384 27405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:06.886404 27405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:06.886420 27405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:06.886435 27405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:06.888077 27405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:06.888348 27405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:06.888571 27405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:06.898751 27417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.898756 27420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:06.898751 27416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:06.898779 27405 server_base.cc:1034] running on GCE node
I20260811 02:40:06.899307 27405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:06.899652 27405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:06.900836 27405 hybrid_clock.cc:648] HybridClock initialized: now 1786416006900822 us; error 35 us; skew 500 ppm
Aug 11 02:40:06 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:06.903648 27405 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:06.904170 27405 webserver.cc:460] Webserver started at http://127.2.251.62:46311/ using document root <none> and password file <none>
I20260811 02:40:06.904417 27405 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:06.904487 27405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:06.906066 27405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:06.906945 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:06.907276 27405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:06.907397 27405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "8911d4bce42643e8a9a01d32602d5e46"
format_stamp: "Formatted at 2026-08-11 02:40:05 on dist-test-slave-cvwc"
server_key: "19ad009a7ba1c7c9c6398bd11b2d81a1"
server_key_iv: "eff59ca8a968f8bb95194edc9b4b8bc1"
server_key_version: "encryptionkey@0"
I20260811 02:40:06.907846 27405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:06.925135 27405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:06.927685 27405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:06.927856 27405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:06.932845 27405 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40427
I20260811 02:40:06.932987 27526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40427 every 8 connection(s)
I20260811 02:40:06.933616 27405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:06.935592 27528 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:40:06.936348 27528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap starting.
I20260811 02:40:06.942008 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 27405
I20260811 02:40:06.942454 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:42551
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:40427
--webserver_port=46681
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:06.945324 27528 log.cc:824] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:06.947542 27528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:40:06.947904 27528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap complete.
I20260811 02:40:06.950229 27528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:06.950481 27528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8911d4bce42643e8a9a01d32602d5e46, State: Initialized, Role: FOLLOWER
I20260811 02:40:06.950645 27528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:06.950729 27528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:40:06.950791 27528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:40:06.950852 27528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 1 FOLLOWER]: Advancing to term 2
I20260811 02:40:06.951953 27528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:06.952064 27528 leader_election.cc:302] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8911d4bce42643e8a9a01d32602d5e46; no voters:
I20260811 02:40:06.952261 27528 leader_election.cc:288] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 02:40:06.952325 27538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 02:40:06.952459 27538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 LEADER]: Becoming Leader. State: Replica: 8911d4bce42643e8a9a01d32602d5e46, State: Running, Role: LEADER
I20260811 02:40:06.952556 27528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:06.952613 27538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:06.952948 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8911d4bce42643e8a9a01d32602d5e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } } }
I20260811 02:40:06.953025 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:06.953238 27541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8911d4bce42643e8a9a01d32602d5e46. Latest consensus state: current_term: 2 leader_uuid: "8911d4bce42643e8a9a01d32602d5e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } } }
I20260811 02:40:06.953370 27541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:06.954398 27562 catalog_manager.cc:1259] Loaded cluster ID: b50b6e0b021c4b3bacd404d5f4954b2b
I20260811 02:40:06.954442 27562 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: loading cluster ID for follower catalog manager: success
W20260811 02:40:06.954631 27562 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260811 02:40:06.954664 27562 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260811 02:40:06.954708 27547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:06.955137 27547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:40:06.955221 27547 catalog_manager.cc:1259] Loaded cluster ID: b50b6e0b021c4b3bacd404d5f4954b2b
I20260811 02:40:06.955242 27547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:40:06.955412 27547 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 02:40:06.960541 27547 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 02:40:06.963394 27547 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:40:06.963636 27547 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Loaded TSK: 0
I20260811 02:40:06.963862 27547 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 02:40:07.043763 27530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:07.044147 27530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:07.044235 27530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:07.046361 27530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:07.046495 27530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:07.046586 27530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:40:07.050174 27530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:42551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=46681
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:07.050769 27530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:07.051190 27530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:07.062369 27574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.062474 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.062383 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.064370 27530 server_base.cc:1034] running on GCE node
I20260811 02:40:07.064649 27530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:07.065006 27530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:07.066157 27530 hybrid_clock.cc:648] HybridClock initialized: now 1786416007066119 us; error 49 us; skew 500 ppm
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:07.071803 27530 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:40:07.074380 27530 webserver.cc:460] Webserver started at http://127.2.251.1:46681/ using document root <none> and password file <none>
I20260811 02:40:07.074681 27530 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:07.074769 27530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:07.077129 27530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:40:07.078122 27593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.078336 27530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.078476 27530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "00fa813265054041b4c90a3146983f81"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "93be7f7fec47cdd92095650c6067e024"
server_key_iv: "48842e4f1036266be755af9edbd76120"
server_key_version: "encryptionkey@0"
I20260811 02:40:07.078986 27530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:07.091192 27530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:07.101997 27530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:07.102171 27530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:07.102478 27530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:07.103037 27610 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 02:40:07.104655 27530 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 02:40:07.104712 27530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 02:40:07.104748 27530 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 02:40:07.105631 27530 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 02:40:07.105674 27530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:07.105870 27610 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Bootstrap starting.
I20260811 02:40:07.115672 27530 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:42551
I20260811 02:40:07.115767 27800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:42551 every 8 connection(s)
I20260811 02:40:07.116158 27530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:40:07.117508 27610 log.cc:824] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:07.119715 27610 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:40:07.120019 27610 tablet_bootstrap.cc:492] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Bootstrap complete.
I20260811 02:40:07.120841 27610 ts_tablet_manager.cc:1397] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260811 02:40:07.122089 27610 raft_consensus.cc:348] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:07.122381 27610 raft_consensus.cc:729] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fa813265054041b4c90a3146983f81, State: Initialized, Role: FOLLOWER
I20260811 02:40:07.122529 27610 consensus_queue.cc:260] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } } peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } }
I20260811 02:40:07.122825 27610 ts_tablet_manager.cc:1428] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:07.124783 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 27530
I20260811 02:40:07.125102 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:36753
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:40427
--webserver_port=36507
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:07.128320 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.117090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:47199 (local address 127.2.251.62:40427)
0811 02:40:07.117283 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.117291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.117351 (+ 60us) server_negotiation.cc:407] Connection header received
0811 02:40:07.118072 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.118077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.118129 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.118245 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:07.118904 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.119768 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.120833 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.121249 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.121365 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.124155 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:07.124161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:07.124162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:07.124179 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:07.126229 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.126822 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.126826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.126826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.126934 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.127244 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.127245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.127245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.127530 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:07.127608 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.128023 (+ 415us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.128153 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260811 02:40:07.128353 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.116961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:47199)
0811 02:40:07.117156 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.117202 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.117864 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.118283 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.118287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.118594 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:07.118779 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.118785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.119928 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.119929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.120747 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.120750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.121349 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.121351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.121485 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.122091 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:07.122099 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:07.123900 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:07.126367 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:07.126377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:07.126379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:07.126684 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:07.127001 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:07.127002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:07.127002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:07.127150 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:07.127608 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:07.127614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:07.127789 (+ 175us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.127929 (+ 140us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.128138 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260811 02:40:07.129274 27802 heartbeater.cc:347] Connected to a master server at 127.2.251.62:40427
I20260811 02:40:07.129395 27802 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:07.129599 27802 heartbeater.cc:511] Master 127.2.251.62:40427 requested a full tablet report, sending...
I20260811 02:40:07.130698 27471 ts_manager.cc:194] Registered new tserver with Master: 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551)
I20260811 02:40:07.132014 27471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:47199
I20260811 02:40:07.133054 27802 heartbeater.cc:503] Master 127.2.251.62:40427 was elected leader, sending a full tablet report...
I20260811 02:40:07.137838 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.131585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42551 (local address 127.0.0.1:44090)
0811 02:40:07.133846 (+ 2261us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.133895 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.133971 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.134310 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.134313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.134316 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:07.134381 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.134382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.135603 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.135604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.136538 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.136542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.137515 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.137516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.137596 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.137608 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.137666 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.137742 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2219,"thread_start_us":93,"threads_started":1}
I20260811 02:40:07.137851 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.131670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.2.251.1:42551)
0811 02:40:07.134119 (+ 2449us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.134124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.134134 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:40:07.134169 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.134172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.134188 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.134271 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:07.134460 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.135513 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.136622 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.137413 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.137484 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.137550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.137698 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.137768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2406,"thread_start_us":90,"threads_started":1}
I20260811 02:40:07.138302 27657 tablet_service.cc:1514] Processing DeleteTablet for tablet 490eea2c622f44f1b928104377cf4c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:06 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:44090
I20260811 02:40:07.138854 27813 tablet_replica.cc:330] stopping tablet replica
I20260811 02:40:07.138979 27813 raft_consensus.cc:2229] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:40:07.139052 27813 raft_consensus.cc:2258] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:07.139405 27813 ts_tablet_manager.cc:1905] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:07.141875 27813 ts_tablet_manager.cc:1918] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:07.141968 27813 log.cc:1196] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/490eea2c622f44f1b928104377cf4c7d
I20260811 02:40:07.142200 27813 ts_tablet_manager.cc:1939] T 490eea2c622f44f1b928104377cf4c7d P 00fa813265054041b4c90a3146983f81: Deleting consensus metadata
I20260811 02:40:07.142980 27440 catalog_manager.cc:4841] TS 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551): tablet 490eea2c622f44f1b928104377cf4c7d (table test-table [id=0a5792c138a94c07937efff2930f8cab]) successfully deleted
W20260811 02:40:07.219046 27810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:07.219276 27810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:07.219303 27810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:07.220547 27810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:07.220594 27810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:07.220628 27810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:40:07.222204 27810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=36507
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:07.222450 27810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:07.222679 27810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:07.228277 27821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.228305 27825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.228452 27810 server_base.cc:1034] running on GCE node
W20260811 02:40:07.228319 27831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.228700 27810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:07.228978 27810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:07.230176 27810 hybrid_clock.cc:648] HybridClock initialized: now 1786416007230156 us; error 35 us; skew 500 ppm
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:07.232972 27810 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:40:07.233500 27810 webserver.cc:460] Webserver started at http://127.2.251.2:36507/ using document root <none> and password file <none>
I20260811 02:40:07.233860 27810 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:07.233942 27810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:07.235446 27810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.236138 27842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.236397 27810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.236495 27810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "dbf82d620d17486a9757a7a97b5f767f"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "7cf40692de6bba9eed9d8eafd01fb8b9"
server_key_iv: "5fcb165475e93f426380680aa3a128aa"
server_key_version: "encryptionkey@0"
I20260811 02:40:07.236903 27810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:07.260165 27810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:07.262710 27810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:07.262871 27810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:07.263135 27810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:07.263473 27810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:07.263652 27810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.263742 27810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:07.263769 27810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.270552 27810 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:36753
I20260811 02:40:07.270622 28019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:36753 every 8 connection(s)
I20260811 02:40:07.271010 27810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:40:07.273386 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 27810
I20260811 02:40:07.273697 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:34441
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:40427
--webserver_port=41213
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:07.282040 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.271792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:43745 (local address 127.2.251.62:40427)
0811 02:40:07.271862 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.271864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.272076 (+ 212us) server_negotiation.cc:407] Connection header received
0811 02:40:07.272630 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.272631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.272644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.272713 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:07.273986 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.274517 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.275865 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.276174 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.276260 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.278926 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:07.278928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:07.278928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:07.278936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:07.280298 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.280828 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.280830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.280830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.280882 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.281210 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.281212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.281212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.281365 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:07.281431 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.281872 (+ 441us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.281961 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 02:40:07.282413 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.271778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:43745)
0811 02:40:07.271950 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.271994 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.272509 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.272787 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.272790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.273515 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:07.273867 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.273876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.274620 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.274621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.275745 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.275749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.276279 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.276280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.276396 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.276984 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:07.276996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:07.278796 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:07.280392 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:07.280402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:07.280406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:07.280738 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:07.280950 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:07.280951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:07.280951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:07.281113 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:07.281443 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:07.281452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:07.281648 (+ 196us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.281819 (+ 171us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.282026 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260811 02:40:07.283077 28020 heartbeater.cc:347] Connected to a master server at 127.2.251.62:40427
I20260811 02:40:07.283203 28020 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:07.283442 28020 heartbeater.cc:511] Master 127.2.251.62:40427 requested a full tablet report, sending...
I20260811 02:40:07.283962 27471 ts_manager.cc:194] Registered new tserver with Master: dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
I20260811 02:40:07.284373 27471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:43745
W20260811 02:40:07.365309 28023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:07.365551 28023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:07.365577 28023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:07.366988 28023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:07.367053 28023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:07.367084 28023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:40:07.368808 28023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:34441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=41213
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:07.369088 28023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:07.369495 28023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:07.375455 28039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.375458 28033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.375458 28035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.375792 28023 server_base.cc:1034] running on GCE node
I20260811 02:40:07.376022 28023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:07.376281 28023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:07.377468 28023 hybrid_clock.cc:648] HybridClock initialized: now 1786416007377452 us; error 33 us; skew 500 ppm
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:07.380532 28023 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:40:07.381007 28023 webserver.cc:460] Webserver started at http://127.2.251.3:41213/ using document root <none> and password file <none>
I20260811 02:40:07.381237 28023 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:07.381295 28023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:07.382695 28023 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.383281 28052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.383579 28023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:07.383773 28023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "2c988bdc99b14194b204c06b24c77b1a"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "db3570ad039e1f318d565f9f78864c94"
server_key_iv: "3bf10bd0cc48220b98981533e94be448"
server_key_version: "encryptionkey@0"
I20260811 02:40:07.384441 28023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:07.391633 28023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:07.394117 28023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:07.394281 28023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:07.394546 28023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:07.394865 28023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:07.394902 28023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.394933 28023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:07.394951 28023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.400929 28023 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:34441
I20260811 02:40:07.400995 28261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:34441 every 8 connection(s)
I20260811 02:40:07.401366 28023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:07.411268 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 28023
I20260811 02:40:07.411278 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.402150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:58567 (local address 127.2.251.62:40427)
0811 02:40:07.402206 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.402209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.402533 (+ 324us) server_negotiation.cc:407] Connection header received
0811 02:40:07.403191 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.403192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.403203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.403260 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:07.404324 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.404831 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.405932 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.406137 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.406209 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.408488 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:07.408490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:07.408490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:07.408499 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:07.409834 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.410244 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.410245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.410246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.410302 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.410540 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.410541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.410541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.410665 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:07.410724 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.411044 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.411176 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:40:07.411667 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.402147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:58567)
0811 02:40:07.402373 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.402413 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.403068 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.403399 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.403403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.404000 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:07.404201 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.404206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.404926 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.404927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.405829 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.405832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.406237 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.406239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.406356 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.406950 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:07.406960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:07.408378 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:07.409923 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:07.409932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:07.409935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:07.410174 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:07.410368 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:07.410369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:07.410369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:07.410483 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:07.410721 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:07.410728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:07.410915 (+ 187us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.411013 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.411184 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260811 02:40:07.412397 28262 heartbeater.cc:347] Connected to a master server at 127.2.251.62:40427
I20260811 02:40:07.412498 28262 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:07.412709 28262 heartbeater.cc:511] Master 127.2.251.62:40427 requested a full tablet report, sending...
I20260811 02:40:07.413219 27471 ts_manager.cc:194] Registered new tserver with Master: 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441)
I20260811 02:40:07.413597 27471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:58567
I20260811 02:40:07.419094 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.411809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.2.251.62:40427)
0811 02:40:07.412057 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.412060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.412117 (+ 57us) server_negotiation.cc:407] Connection header received
0811 02:40:07.412213 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.412214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.412224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.412292 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:07.413076 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.413637 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.414487 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.414675 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.414734 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.416121 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:07.416123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:07.416123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:07.416131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:07.417573 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.418025 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.418026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.418026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.418104 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:07.418381 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:07.418384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:07.418384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:07.418628 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:07.418860 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.418895 (+ 35us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.419001 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260811 02:40:07.420106 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:40:07.423197 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.420645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.2.251.62:40427)
0811 02:40:07.420750 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.420752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.420761 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:40:07.420864 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.420864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.420901 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.420961 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:40:07.421088 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.421586 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.422471 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.422656 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.422717 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.422868 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:40:07.422983 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:40:07.423056 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.423092 (+ 36us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.423112 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 02:40:07.423509 27471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:40:07.425127 27471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:40:07.427716 27657 tablet_service.cc:1467] Processing CreateTablet for tablet 7db7607ee3da4a09850ce99f2874fc9e (DEFAULT_TABLE table=test-table [id=0d12ba96f3e54110a80fa650ea9188b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:07.427977 27657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db7607ee3da4a09850ce99f2874fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:07.429705 27610 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Bootstrap starting.
I20260811 02:40:07.430207 27610 tablet_bootstrap.cc:654] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:07.430717 27610 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: No bootstrap required, opened a new log
I20260811 02:40:07.430775 27610 ts_tablet_manager.cc:1397] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.430891 27610 raft_consensus.cc:348] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.430967 27610 raft_consensus.cc:374] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:07.431015 27610 raft_consensus.cc:729] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fa813265054041b4c90a3146983f81, State: Initialized, Role: FOLLOWER
I20260811 02:40:07.431061 27610 consensus_queue.cc:260] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.431151 27610 ts_tablet_manager.cc:1428] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260811 02:40:07.431670 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.427392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34441 (local address 127.0.0.1:34646)
0811 02:40:07.427446 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.427458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.427495 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.427950 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.427951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.427953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:07.428003 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.428005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.429130 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.429131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.430465 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.430468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.431433 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.431434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.431502 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.431506 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.431550 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.431600 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 02:40:07.432039 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.427508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.2.251.3:34441)
0811 02:40:07.427703 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.427708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.427720 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:40:07.427777 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.427779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.427796 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.428071 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:07.428149 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.429036 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.430561 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.431319 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.431427 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.431893 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.431924 (+ 31us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.431971 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260811 02:40:07.432161 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.427759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.2.251.2:36753)
0811 02:40:07.427944 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.427948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.427957 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:40:07.428002 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.428004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.428019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.428254 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:07.428322 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.429322 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.430900 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.431635 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.431705 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.431767 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.432031 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.432075 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260811 02:40:07.432428 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.427479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:36753 (local address 127.0.0.1:55820)
0811 02:40:07.427742 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.427752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.427805 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.428121 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.428123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.428125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:07.428181 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.428183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.429432 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.429433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.430831 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.430834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.431894 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.431895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.431947 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.431951 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.432003 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.432049 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260811 02:40:07.432848 28142 tablet_service.cc:1467] Processing CreateTablet for tablet 7db7607ee3da4a09850ce99f2874fc9e (DEFAULT_TABLE table=test-table [id=0d12ba96f3e54110a80fa650ea9188b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:07.433290 28142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db7607ee3da4a09850ce99f2874fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:07.433447 27920 tablet_service.cc:1467] Processing CreateTablet for tablet 7db7607ee3da4a09850ce99f2874fc9e (DEFAULT_TABLE table=test-table [id=0d12ba96f3e54110a80fa650ea9188b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:07.433799 27920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db7607ee3da4a09850ce99f2874fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:07.436311 28285 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Bootstrap starting.
I20260811 02:40:07.436872 28287 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Bootstrap starting.
I20260811 02:40:07.437202 28285 tablet_bootstrap.cc:654] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:07.437776 28285 log.cc:824] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:07.437860 28287 tablet_bootstrap.cc:654] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:07.438534 28287 log.cc:824] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:07.438632 28285 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: No bootstrap required, opened a new log
I20260811 02:40:07.438697 28285 ts_tablet_manager.cc:1397] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 02:40:07.439359 28287 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: No bootstrap required, opened a new log
I20260811 02:40:07.439440 28287 ts_tablet_manager.cc:1397] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 02:40:07.440410 28285 raft_consensus.cc:348] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.440572 28285 raft_consensus.cc:374] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:07.440594 28285 raft_consensus.cc:729] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c988bdc99b14194b204c06b24c77b1a, State: Initialized, Role: FOLLOWER
I20260811 02:40:07.440706 28285 consensus_queue.cc:260] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.441030 28285 ts_tablet_manager.cc:1428] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260811 02:40:07.441077 28262 heartbeater.cc:503] Master 127.2.251.62:40427 was elected leader, sending a full tablet report...
I20260811 02:40:07.441197 28287 raft_consensus.cc:348] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.441385 28287 raft_consensus.cc:374] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:07.441421 28287 raft_consensus.cc:729] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf82d620d17486a9757a7a97b5f767f, State: Initialized, Role: FOLLOWER
I20260811 02:40:07.441514 28287 consensus_queue.cc:260] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.441814 28287 ts_tablet_manager.cc:1428] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260811 02:40:07.441875 28020 heartbeater.cc:503] Master 127.2.251.62:40427 was elected leader, sending a full tablet report...
I20260811 02:40:07.521083 28295 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:07.521322 28295 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
W20260811 02:40:07.521682 28021 tablet.cc:2369] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:40:07.521880 28295 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551)
I20260811 02:40:07.525074 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.522072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34441 (local address 127.2.251.2:40833)
0811 02:40:07.522114 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.522128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.522174 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.522438 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.522439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.522441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:07.522511 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.522513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.523323 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.523325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.524127 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.524130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.524791 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.524793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.524907 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.524911 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.524966 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.525007 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":1}
I20260811 02:40:07.525146 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.522222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:40833 (local address 127.2.251.3:34441)
0811 02:40:07.522293 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.522295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.522306 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:40:07.522337 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.522338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.522349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.522417 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:07.522622 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.523244 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.524227 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.524691 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.524749 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.524761 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.525008 (+ 247us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.525062 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 02:40:07.525218 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.522072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42551 (local address 127.2.251.2:34571)
0811 02:40:07.522434 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.522445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.522529 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:07.522693 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:07.522694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:07.522695 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:07.522764 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.522765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.523473 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.523474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.524275 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:07.524278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.524940 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.524941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:07.525027 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.525031 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:40:07.525097 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 02:40:07.525141 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":143,"thread_start_us":153,"threads_started":1}
I20260811 02:40:07.525231 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.522211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:34571 (local address 127.2.251.1:42551)
0811 02:40:07.522272 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.522275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.522510 (+ 235us) server_negotiation.cc:407] Connection header received
0811 02:40:07.522587 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.522588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.522601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.522647 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:07.522830 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.523388 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.524406 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.524812 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.524902 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.524913 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.525122 (+ 209us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.525168 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:40:07.525509 28177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7607ee3da4a09850ce99f2874fc9e" candidate_uuid: "dbf82d620d17486a9757a7a97b5f767f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c988bdc99b14194b204c06b24c77b1a" is_pre_election: true
I20260811 02:40:07.525527 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7607ee3da4a09850ce99f2874fc9e" candidate_uuid: "dbf82d620d17486a9757a7a97b5f767f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fa813265054041b4c90a3146983f81" is_pre_election: true
I20260811 02:40:07.525684 28177 raft_consensus.cc:2454] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf82d620d17486a9757a7a97b5f767f in term 0.
I20260811 02:40:07.525684 27710 raft_consensus.cc:2454] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf82d620d17486a9757a7a97b5f767f in term 0.
I20260811 02:40:07.525971 27847 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fa813265054041b4c90a3146983f81, dbf82d620d17486a9757a7a97b5f767f; no voters:
I20260811 02:40:07.526104 28295 raft_consensus.cc:2781] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:40:07.526173 28295 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:40:07.526192 28295 raft_consensus.cc:3037] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:07.526994 28295 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.527143 28295 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [CANDIDATE]: Term 1 election: Requested vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551)
I20260811 02:40:07.527357 28177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7607ee3da4a09850ce99f2874fc9e" candidate_uuid: "dbf82d620d17486a9757a7a97b5f767f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c988bdc99b14194b204c06b24c77b1a"
I20260811 02:40:07.527470 28177 raft_consensus.cc:3037] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:07.527550 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7607ee3da4a09850ce99f2874fc9e" candidate_uuid: "dbf82d620d17486a9757a7a97b5f767f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00fa813265054041b4c90a3146983f81"
I20260811 02:40:07.527633 27710 raft_consensus.cc:3037] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:07.528483 28177 raft_consensus.cc:2454] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf82d620d17486a9757a7a97b5f767f in term 1.
I20260811 02:40:07.528805 27851 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f; no voters:
I20260811 02:40:07.528972 27710 raft_consensus.cc:2454] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf82d620d17486a9757a7a97b5f767f in term 1.
I20260811 02:40:07.529001 28295 raft_consensus.cc:2781] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:07.529196 28295 raft_consensus.cc:686] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 1 LEADER]: Becoming Leader. State: Replica: dbf82d620d17486a9757a7a97b5f767f, State: Running, Role: LEADER
I20260811 02:40:07.529479 28295 consensus_queue.cc:237] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.530695 27467 catalog_manager.cc:5495] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf82d620d17486a9757a7a97b5f767f (127.2.251.2). New cstate: current_term: 1 leader_uuid: "dbf82d620d17486a9757a7a97b5f767f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } health_report { overall_health: UNKNOWN } } }
I20260811 02:40:07.551184 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.548875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.2.251.2:36753)
0811 02:40:07.548943 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:07.548945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:07.548952 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:40:07.548989 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:07.548990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:07.549010 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:07.549071 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:40:07.549189 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.549685 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.550447 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:07.550635 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:07.550691 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:07.550834 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:40:07.550953 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:40:07.551029 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:07.551051 (+ 22us) server_negotiation.cc:299] Negotiation successful
0811 02:40:07.551075 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:40:07.553269 28177 raft_consensus.cc:1261] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Refusing update from remote peer dbf82d620d17486a9757a7a97b5f767f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:07.553288 27710 raft_consensus.cc:1261] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Refusing update from remote peer dbf82d620d17486a9757a7a97b5f767f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:07.553694 28297 consensus_queue.cc:1035] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:07.553931 28295 consensus_queue.cc:1035] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:07.556178 28304 mvcc.cc:205] Tried to move back new op lower bound from 7317159966935023616 to 7317159966841692160. Current Snapshot: MvccSnapshot[applied={T|T < 7317159966935023616}]
I20260811 02:40:07.556437 28306 mvcc.cc:205] Tried to move back new op lower bound from 7317159966935023616 to 7317159966841692160. Current Snapshot: MvccSnapshot[applied={T|T < 7317159966935023616}]
I20260811 02:40:07.556671 28305 mvcc.cc:205] Tried to move back new op lower bound from 7317159966935023616 to 7317159966841692160. Current Snapshot: MvccSnapshot[applied={T|T < 7317159966935023616}]
I20260811 02:40:07.559381 27471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:07.559481 27471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:07.560194 27471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Sending DeleteTablet for 3 replicas of tablet 7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:07.560510 27920 tablet_service.cc:1514] Processing DeleteTablet for tablet 7db7607ee3da4a09850ce99f2874fc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:07 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:55820
I20260811 02:40:07.560617 28142 tablet_service.cc:1514] Processing DeleteTablet for tablet 7db7607ee3da4a09850ce99f2874fc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:07 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:34646
I20260811 02:40:07.560770 28319 tablet_replica.cc:330] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: stopping tablet replica
I20260811 02:40:07.560774 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 27530
I20260811 02:40:07.560791 27657 tablet_service.cc:1514] Processing DeleteTablet for tablet 7db7607ee3da4a09850ce99f2874fc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:07 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:44090
I20260811 02:40:07.560899 28319 raft_consensus.cc:2229] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:40:07.561033 28319 raft_consensus.cc:2258] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:07.569393 28319 ts_tablet_manager.cc:1905] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:07.571686 28319 ts_tablet_manager.cc:1918] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:07.571787 28319 log.cc:1196] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:07.572084 28319 ts_tablet_manager.cc:1939] T 7db7607ee3da4a09850ce99f2874fc9e P dbf82d620d17486a9757a7a97b5f767f: Deleting consensus metadata
I20260811 02:40:07.572583 27436 catalog_manager.cc:4841] TS dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): tablet 7db7607ee3da4a09850ce99f2874fc9e (table test-table [id=0d12ba96f3e54110a80fa650ea9188b0]) successfully deleted
I20260811 02:40:07.580365 28321 tablet_replica.cc:330] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: stopping tablet replica
I20260811 02:40:07.580547 28321 raft_consensus.cc:2229] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:40:07.580687 28321 raft_consensus.cc:2258] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:07.581228 28321 ts_tablet_manager.cc:1905] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:07.582903 28321 ts_tablet_manager.cc:1918] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:07.582988 28321 log.cc:1196] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:07.583236 28321 ts_tablet_manager.cc:1939] T 7db7607ee3da4a09850ce99f2874fc9e P 2c988bdc99b14194b204c06b24c77b1a: Deleting consensus metadata
I20260811 02:40:07.583752 27444 catalog_manager.cc:4841] TS 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): tablet 7db7607ee3da4a09850ce99f2874fc9e (table test-table [id=0d12ba96f3e54110a80fa650ea9188b0]) successfully deleted
W20260811 02:40:07.586254 27440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:42551 (error 108)
I20260811 02:40:07.586793 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:07.586670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42551 (local address 127.0.0.1:44106)
0811 02:40:07.586717 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.586755 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 02:40:07.586831 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 27810
W20260811 02:40:07.599417 27440 catalog_manager.cc:4584] TS 00fa813265054041b4c90a3146983f81 (127.2.251.1:42551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7db7607ee3da4a09850ce99f2874fc9e: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42551: connect: Connection refused (error 111)
I20260811 02:40:07.605121 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 28023
I20260811 02:40:07.622690 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 27405
I20260811 02:40:07.643919 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40427
--webserver_interface=127.2.251.62
--webserver_port=46311
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:07.732338 28325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:07.732592 28325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:07.732616 28325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:07.733942 28325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:07.734000 28325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:07.734014 28325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:07.734026 28325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:07.734037 28325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:07.735718 28325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:07.736018 28325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:07.736270 28325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:07.741845 28336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.741832 28331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.741830 28325 server_base.cc:1034] running on GCE node
W20260811 02:40:07.741827 28333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.742722 28325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:07.743216 28325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:07.744421 28325 hybrid_clock.cc:648] HybridClock initialized: now 1786416007744380 us; error 60 us; skew 500 ppm
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:07.747543 28325 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:07.748133 28325 webserver.cc:460] Webserver started at http://127.2.251.62:46311/ using document root <none> and password file <none>
I20260811 02:40:07.748375 28325 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:07.748433 28325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:07.749897 28325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.751135 28347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.751410 28325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.751639 28325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "8911d4bce42643e8a9a01d32602d5e46"
format_stamp: "Formatted at 2026-08-11 02:40:05 on dist-test-slave-cvwc"
server_key: "19ad009a7ba1c7c9c6398bd11b2d81a1"
server_key_iv: "eff59ca8a968f8bb95194edc9b4b8bc1"
server_key_version: "encryptionkey@0"
I20260811 02:40:07.752067 28325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:07.765573 28325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:07.768348 28325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:07.768514 28325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:07.773252 28325 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40427
I20260811 02:40:07.773312 28429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40427 every 8 connection(s)
I20260811 02:40:07.773751 28325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:07.775933 28430 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:40:07.776711 28430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap starting.
I20260811 02:40:07.781067 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 28325
I20260811 02:40:07.781394 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:42551
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:40427
--webserver_port=46681
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:07.785892 28430 log.cc:824] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:07.789356 28430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:40:07.789812 28430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46: Bootstrap complete.
I20260811 02:40:07.793208 28430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:07.793596 28430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8911d4bce42643e8a9a01d32602d5e46, State: Initialized, Role: FOLLOWER
I20260811 02:40:07.793809 28430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:07.793910 28430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:40:07.793967 28430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:40:07.794020 28430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 2 FOLLOWER]: Advancing to term 3
I20260811 02:40:07.795549 28430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:07.795679 28430 leader_election.cc:302] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8911d4bce42643e8a9a01d32602d5e46; no voters:
I20260811 02:40:07.795873 28430 leader_election.cc:288] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 02:40:07.795951 28440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 02:40:07.796134 28440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [term 3 LEADER]: Becoming Leader. State: Replica: 8911d4bce42643e8a9a01d32602d5e46, State: Running, Role: LEADER
I20260811 02:40:07.796170 28430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:07.796280 28440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } }
I20260811 02:40:07.796686 28441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8911d4bce42643e8a9a01d32602d5e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } } }
I20260811 02:40:07.796693 28442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8911d4bce42643e8a9a01d32602d5e46. Latest consensus state: current_term: 3 leader_uuid: "8911d4bce42643e8a9a01d32602d5e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8911d4bce42643e8a9a01d32602d5e46" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40427 } } }
I20260811 02:40:07.796779 28441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:07.796803 28442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8911d4bce42643e8a9a01d32602d5e46 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:07.797439 28451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:07.798355 28451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:40:07.798487 28451 catalog_manager.cc:1259] Loaded cluster ID: b50b6e0b021c4b3bacd404d5f4954b2b
I20260811 02:40:07.798524 28451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 02:40:07.798861 28451 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe84c6ac9ad google::LogMessage::Fail() at ??:0
@ 0x7fe84c6af88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe84c6ac4de google::LogMessage::Flush() at ??:0
@ 0x7fe84c6ada19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe8545cd0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe8545d68b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe84e57a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe84e56f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe84f3d36db start_thread at ??:0
@ 0x7fe84b1aa61f clone at ??:0
W20260811 02:40:07.889302 28436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:07.889539 28436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:07.889567 28436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:07.890928 28436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:07.890990 28436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:07.891019 28436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:40:07.892653 28436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:42551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=46681
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:07.892954 28436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:07.893199 28436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:07.899799 28500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:07.899859 28495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.900041 28436 server_base.cc:1034] running on GCE node
W20260811 02:40:07.899827 28496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:07.900339 28436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:07.900624 28436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:07.901872 28436 hybrid_clock.cc:648] HybridClock initialized: now 1786416007901853 us; error 48 us; skew 500 ppm
Aug 11 02:40:07 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:07.904773 28436 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:40:07.905493 28436 webserver.cc:460] Webserver started at http://127.2.251.1:46681/ using document root <none> and password file <none>
I20260811 02:40:07.905844 28436 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:07.905927 28436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:07.907925 28436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:40:07.908911 28510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:07.909111 28436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:07.909221 28436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "00fa813265054041b4c90a3146983f81"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "93be7f7fec47cdd92095650c6067e024"
server_key_iv: "48842e4f1036266be755af9edbd76120"
server_key_version: "encryptionkey@0"
I20260811 02:40:07.909795 28436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:07.959004 28436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:07.961948 28436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:07.962147 28436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:07.962458 28436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:07.963057 28519 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 02:40:07.964586 28436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 02:40:07.964648 28436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 02:40:07.964685 28436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 02:40:07.965622 28436 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 02:40:07.965669 28436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:07.965766 28519 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Bootstrap starting.
I20260811 02:40:07.973156 28436 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:42551
I20260811 02:40:07.973219 28690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:42551 every 8 connection(s)
I20260811 02:40:07.973649 28436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:40:07.976014 28519 log.cc:824] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:07.978379 28519 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:40:07.978650 28519 tablet_bootstrap.cc:492] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Bootstrap complete.
I20260811 02:40:07.979444 28519 ts_tablet_manager.cc:1397] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260811 02:40:07.979795 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 28436
I20260811 02:40:07.980059 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:36753
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:40427
--webserver_port=36507
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:07.980851 28519 raft_consensus.cc:348] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.981163 28519 raft_consensus.cc:729] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fa813265054041b4c90a3146983f81, State: Initialized, Role: FOLLOWER
I20260811 02:40:07.981334 28519 consensus_queue.cc:260] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:07.981628 28519 ts_tablet_manager.cc:1428] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260811 02:40:08.009904 28695 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:40:07.974461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:54643)
0811 02:40:07.974759 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:07.974806 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:07.975461 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:08.009600 (+ 34139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
W20260811 02:40:08.010187 28691 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:40427 (0 consecutive failures): Network error: Failed to ping master at 127.2.251.62:40427: Client connection negotiation failed: client connection to 127.2.251.62:40427: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 02:40:08.010571 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.010419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:38167)
0811 02:40:08.010476 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.010537 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 02:40:08.010944 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.010850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:56563)
0811 02:40:08.010891 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.010921 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260811 02:40:08.011245 28691 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:40:08.070976 28699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:08.071211 28699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:08.071238 28699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:08.072564 28699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:08.072625 28699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:08.072654 28699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:40:08.074421 28699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=36507
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:08.074666 28699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:08.074895 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:08.080226 28707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:08.080243 28708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:08.080235 28710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:08.080623 28699 server_base.cc:1034] running on GCE node
I20260811 02:40:08.080837 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:08.081089 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:08.082254 28699 hybrid_clock.cc:648] HybridClock initialized: now 1786416008082240 us; error 30 us; skew 500 ppm
Aug 11 02:40:08 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:08.084944 28699 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:40:08.085453 28699 webserver.cc:460] Webserver started at http://127.2.251.2:36507/ using document root <none> and password file <none>
I20260811 02:40:08.085672 28699 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:08.085763 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:08.087404 28699 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:08.088124 28723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:08.088450 28699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:08.088562 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "dbf82d620d17486a9757a7a97b5f767f"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "7cf40692de6bba9eed9d8eafd01fb8b9"
server_key_iv: "5fcb165475e93f426380680aa3a128aa"
server_key_version: "encryptionkey@0"
I20260811 02:40:08.088956 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:08.105700 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:08.108080 28699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:08.108243 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:08.108518 28699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:08.108875 28699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:08.108918 28699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:08.108942 28699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:08.108953 28699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:08.114995 28699 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:36753
I20260811 02:40:08.115063 28918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:36753 every 8 connection(s)
I20260811 02:40:08.115412 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:40:08.116546 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.116110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:39555)
0811 02:40:08.116307 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.116394 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
W20260811 02:40:08.116995 28919 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:40427 (0 consecutive failures): Network error: Failed to ping master at 127.2.251.62:40427: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
I20260811 02:40:08.117017 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 28699
I20260811 02:40:08.117249 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:34441
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:40427
--webserver_port=41213
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:08.117578 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.117253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:54681)
0811 02:40:08.117387 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.117517 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 02:40:08.118144 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.117942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:39837)
0811 02:40:08.118058 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.118103 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260811 02:40:08.118323 28919 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:40:08.207304 28923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:08.207554 28923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:08.207590 28923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:08.208784 28923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:08.208830 28923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:08.208869 28923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:40:08.210534 28923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:34441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=41213
--tserver_master_addrs=127.2.251.62:40427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:08.210816 28923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:08.211045 28923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:08.217962 28935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:08.217962 28940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:08.218362 28937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:08.218282 28923 server_base.cc:1034] running on GCE node
I20260811 02:40:08.218811 28923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:08.219135 28923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:08.220317 28923 hybrid_clock.cc:648] HybridClock initialized: now 1786416008220293 us; error 42 us; skew 500 ppm
Aug 11 02:40:08 dist-test-slave-cvwc krb5kdc[26493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:08.223241 28923 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:40:08.223734 28923 webserver.cc:460] Webserver started at http://127.2.251.3:41213/ using document root <none> and password file <none>
I20260811 02:40:08.223976 28923 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:08.224030 28923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:08.225606 28923 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:40:08.226365 28955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:08.226615 28923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:08.226729 28923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "2c988bdc99b14194b204c06b24c77b1a"
format_stamp: "Formatted at 2026-08-11 02:40:06 on dist-test-slave-cvwc"
server_key: "db3570ad039e1f318d565f9f78864c94"
server_key_iv: "3bf10bd0cc48220b98981533e94be448"
server_key_version: "encryptionkey@0"
I20260811 02:40:08.227123 28923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:08.239781 28923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:08.242192 28923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:08.242340 28923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:08.242592 28923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:08.242933 28923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:08.242969 28923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:08.243000 28923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:08.243023 28923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:08.249106 28923 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:34441
I20260811 02:40:08.249192 29143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:34441 every 8 connection(s)
I20260811 02:40:08.249547 28923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:40:08.250746 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.250315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:41069)
0811 02:40:08.250513 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.250597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
W20260811 02:40:08.251224 29145 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:40427 (0 consecutive failures): Network error: Failed to ping master at 127.2.251.62:40427: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
I20260811 02:40:08.251518 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.251429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:42275)
0811 02:40:08.251471 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.251499 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260811 02:40:08.251828 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:08.251750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:51689)
0811 02:40:08.251786 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:08.251808 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260811 02:40:08.251948 29145 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 02:40:08.254890 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 28923
I20260811 02:40:09.012313 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.011876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:43991)
0811 02:40:09.012155 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:09.012227 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":155,"threads_started":1}
I20260811 02:40:09.119017 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.118727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:50279)
0811 02:40:09.118888 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:09.118943 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260811 02:40:09.252866 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.252559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:55437)
0811 02:40:09.252751 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:09.252802 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260811 02:40:09.315019 29160 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:09.315165 29160 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:09.315543 29160 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
Aug 11 02:40:09 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.3@KRBTEST.COM
Aug 11 02:40:09 dist-test-slave-cvwc krb5kdc[26493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416007, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.2@KRBTEST.COM
I20260811 02:40:09.327483 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.315971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:50769 (local address 127.2.251.2:36753)
0811 02:40:09.316111 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:09.316121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:09.316150 (+ 29us) server_negotiation.cc:407] Connection header received
0811 02:40:09.316619 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:09.316622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:09.316660 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:09.316796 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:09.317621 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.318624 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.319754 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.320161 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.320395 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:09.323957 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:09.323962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:09.323962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:09.323974 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:09.325632 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:09.326226 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:09.326230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:09.326231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:09.326326 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:09.326584 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:09.326585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:09.326586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:09.326852 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:09.326927 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:09.327225 (+ 298us) server_negotiation.cc:299] Negotiation successful
0811 02:40:09.327362 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260811 02:40:09.327497 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.315869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:48663 (local address 127.2.251.3:34441)
0811 02:40:09.316028 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:09.316041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:09.316074 (+ 33us) server_negotiation.cc:407] Connection header received
0811 02:40:09.316561 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:09.316564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:09.316609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:09.316725 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:09.317614 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.318688 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.319738 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.320784 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.320947 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:09.323628 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:09.323634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:09.323634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:09.323649 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:09.325642 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:09.326227 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:09.326230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:09.326231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:09.326327 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:09.326587 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:09.326588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:09.326589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:09.326847 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:09.326920 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:09.327228 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 02:40:09.327365 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260811 02:40:09.327484 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.315773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:36753 (local address 127.2.251.1:50769)
0811 02:40:09.315911 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:09.315922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:09.315987 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:09.316820 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:09.316821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:09.317271 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:09.317430 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:09.317437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.318762 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.318765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:09.319644 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:09.319646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.320416 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.320419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:09.320623 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:09.321358 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:09.321378 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:09.323817 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:09.325760 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:09.325771 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:09.325774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:09.326141 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:09.326392 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:09.326393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:09.326393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:09.326520 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:09.326926 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:09.326927 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:09.327060 (+ 133us) client_negotiation.cc:769] Sending connection context
0811 02:40:09.327155 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 02:40:09.327378 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":75,"thread_start_us":59,"threads_started":1}
I20260811 02:40:09.327577 29155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:09.315758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:34441 (local address 127.2.251.1:48663)
0811 02:40:09.315812 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:09.315829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:09.315897 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:09.316810 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:09.316813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:09.317209 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:09.317407 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:09.317414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.318841 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.318842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:09.319642 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:09.319645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:09.320915 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:09.320916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:09.321016 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:09.321466 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:09.321471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:09.323478 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:09.325788 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:09.325790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:09.325790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:09.326141 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:09.326395 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:09.326396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:09.326396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:09.326521 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:09.326918 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:09.326925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:09.327061 (+ 136us) client_negotiation.cc:769] Sending connection context
0811 02:40:09.327157 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 02:40:09.327457 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":151}
W20260811 02:40:09.328505 28514 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
W20260811 02:40:09.328958 28512 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:09.329025 28512 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fa813265054041b4c90a3146983f81; no voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f
I20260811 02:40:09.329133 29160 raft_consensus.cc:2726] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:40:10.013221 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:10.012925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:42525)
0811 02:40:10.013113 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:10.013161 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260811 02:40:10.119886 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:10.119633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:49477)
0811 02:40:10.119788 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:10.119830 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260811 02:40:10.253897 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:10.253512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:42697)
0811 02:40:10.253795 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:10.253846 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":86,"threads_started":1}
I20260811 02:40:11.014230 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:11.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:55719)
0811 02:40:11.014112 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:11.014162 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260811 02:40:11.064275 29175 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:11.064395 29175 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:11.064622 29175 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
W20260811 02:40:11.065083 28514 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
W20260811 02:40:11.065372 28512 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:11.065502 28512 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fa813265054041b4c90a3146983f81; no voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f
I20260811 02:40:11.065605 29175 raft_consensus.cc:2726] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:40:11.120707 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:11.120410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:57039)
0811 02:40:11.120598 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:11.120643 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
I20260811 02:40:11.255120 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:11.254797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:52945)
0811 02:40:11.255003 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:11.255061 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260811 02:40:12.015141 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:12.014793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:38133)
0811 02:40:12.014989 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:12.015064 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260811 02:40:12.121626 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:12.121348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:44767)
0811 02:40:12.121519 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:12.121571 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260811 02:40:12.256399 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:12.256049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:36841)
0811 02:40:12.256259 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:12.256320 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260811 02:40:13.016191 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:13.015915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:54573)
0811 02:40:13.016087 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:13.016133 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260811 02:40:13.122522 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:13.122178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:53961)
0811 02:40:13.122402 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:13.122469 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260811 02:40:13.257623 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:13.257306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:41627)
0811 02:40:13.257515 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:13.257568 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20260811 02:40:13.456440 29188 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:13.456622 29188 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:13.457020 29188 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
W20260811 02:40:13.457492 28512 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
W20260811 02:40:13.457640 28514 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:13.457798 28514 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fa813265054041b4c90a3146983f81; no voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f
I20260811 02:40:13.457885 29188 raft_consensus.cc:2726] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:40:14.017746 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:14.016987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:37935)
0811 02:40:14.017415 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:14.017453 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":93,"threads_started":1}
I20260811 02:40:14.123441 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:14.123127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:43799)
0811 02:40:14.123333 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:14.123382 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260811 02:40:14.258487 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:14.258209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:34471)
0811 02:40:14.258372 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:14.258424 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260811 02:40:15.018911 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:15.018563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:48449)
0811 02:40:15.018784 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:15.018836 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
I20260811 02:40:15.124343 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:15.124067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:57661)
0811 02:40:15.124236 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:15.124289 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260811 02:40:15.259570 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:15.259151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:36823)
0811 02:40:15.259380 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:15.259456 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260811 02:40:16.020129 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:16.019790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:45387)
0811 02:40:16.019969 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:16.020017 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260811 02:40:16.125373 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:16.125052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:51899)
0811 02:40:16.125251 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:16.125306 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260811 02:40:16.260833 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:16.260449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:39603)
0811 02:40:16.260701 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:16.260775 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":131,"threads_started":1}
I20260811 02:40:16.976171 29210 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:16.976313 29210 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:16.976673 29210 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
W20260811 02:40:16.977147 28512 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
W20260811 02:40:16.977228 28514 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:16.977283 28514 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fa813265054041b4c90a3146983f81; no voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f
I20260811 02:40:16.977357 29210 raft_consensus.cc:2726] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:40:17.021064 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:17.020803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:54041)
0811 02:40:17.020970 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:17.021011 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260811 02:40:17.126438 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:17.126137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:35375)
0811 02:40:17.126293 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:17.126336 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260811 02:40:17.261960 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:17.261502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:54123)
0811 02:40:17.261818 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:17.261879 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":250,"threads_started":1}
I20260811 02:40:18.022104 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:18.021860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:35089)
0811 02:40:18.021999 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:18.022048 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260811 02:40:18.127512 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:18.127202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:56323)
0811 02:40:18.127401 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:18.127454 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260811 02:40:18.263032 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:18.262652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:43061)
0811 02:40:18.262882 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:18.262940 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20260811 02:40:19.023252 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:19.022887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:55339)
0811 02:40:19.023120 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:19.023193 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
I20260811 02:40:19.128614 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:19.128151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:40037)
0811 02:40:19.128417 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:19.128500 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
I20260811 02:40:19.264012 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:19.263674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:35581)
0811 02:40:19.263908 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:19.263958 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260811 02:40:20.024266 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:20.023943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:50961)
0811 02:40:20.024149 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:20.024202 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260811 02:40:20.129612 29224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:20.129310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:45395)
0811 02:40:20.129514 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:20.129555 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":122,"threads_started":1}
I20260811 02:40:20.264900 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:20.264592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:50883)
0811 02:40:20.264800 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:20.264844 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20260811 02:40:21.025189 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:21.024904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:50415)
0811 02:40:21.025089 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:21.025139 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260811 02:40:21.130511 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:21.130201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:56883)
0811 02:40:21.130404 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:21.130455 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260811 02:40:21.266064 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:21.265781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:43999)
0811 02:40:21.265959 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:21.266009 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260811 02:40:22.026099 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:22.025845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:58691)
0811 02:40:22.026006 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:22.026048 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260811 02:40:22.131335 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:22.131047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:48613)
0811 02:40:22.131224 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:22.131278 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260811 02:40:22.266896 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:22.266608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:57731)
0811 02:40:22.266777 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:22.266821 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260811 02:40:22.853273 29240 raft_consensus.cc:482] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:22.853394 29240 raft_consensus.cc:504] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c988bdc99b14194b204c06b24c77b1a" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 34441 } } peers { permanent_uuid: "dbf82d620d17486a9757a7a97b5f767f" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 36753 } } peers { permanent_uuid: "00fa813265054041b4c90a3146983f81" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42551 } }
I20260811 02:40:22.853602 29240 leader_election.cc:288] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441), dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753)
W20260811 02:40:22.854038 28512 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dbf82d620d17486a9757a7a97b5f767f (127.2.251.2:36753): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
W20260811 02:40:22.854125 28514 leader_election.cc:341] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c988bdc99b14194b204c06b24c77b1a (127.2.251.3:34441): Not found: Tablet not found: 7db7607ee3da4a09850ce99f2874fc9e
I20260811 02:40:22.854171 28514 leader_election.cc:302] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00fa813265054041b4c90a3146983f81; no voters: 2c988bdc99b14194b204c06b24c77b1a, dbf82d620d17486a9757a7a97b5f767f
I20260811 02:40:22.854295 29240 raft_consensus.cc:2726] T 7db7607ee3da4a09850ce99f2874fc9e P 00fa813265054041b4c90a3146983f81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:40:23.027182 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:23.026740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.1:53975)
0811 02:40:23.027024 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:23.027108 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
I20260811 02:40:23.132514 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:23.132121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.2:39697)
0811 02:40:23.132363 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:23.132440 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260811 02:40:23.265182 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 28436
I20260811 02:40:23.279207 29246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:23.273899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40427 (local address 127.2.251.3:38765)
0811 02:40:23.278917 (+ 5018us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:23.279107 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4977,"thread_start_us":2105,"threads_started":1}
I20260811 02:40:23.288897 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 28699
I20260811 02:40:23.307109 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 28923
2026-08-11T02:40:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18951 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:40:23 dist-test-slave-cvwc krb5kdc[29250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:40:23 dist-test-slave-cvwc krb5kdc[29250](info): set up 2 sockets
Aug 11 02:40:23 dist-test-slave-cvwc krb5kdc[29250](info): commencing operation
krb5kdc: starting...
W20260811 02:40:24.595597 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:40:24 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:40:24.755185 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38957
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:24.862766 29270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:24.863104 29270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:24.863148 29270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:24.865381 29270 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 02:40:24.865478 29270 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 02:40:24.865500 29270 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 02:40:24.865517 29270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:24.865537 29270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:24.865559 29270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:24.865584 29270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:24.865612 29270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:24.868310 29270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:24.868672 29270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:24.868937 29270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:24.874938 29280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:24.875053 29270 server_base.cc:1034] running on GCE node
W20260811 02:40:24.874958 29290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:24.874958 29281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:24.875685 29270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:24.876027 29270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:24.877200 29270 hybrid_clock.cc:648] HybridClock initialized: now 1786416024877173 us; error 46 us; skew 500 ppm
Aug 11 02:40:24 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416024, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:24.880101 29270 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:24.880612 29270 webserver.cc:460] Webserver started at http://127.2.251.62:46731/ using document root <none> and password file <none>
I20260811 02:40:24.880862 29270 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:24.880905 29270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:24.881006 29270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:24.882121 29270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "933ffd70c9a24a78a4bae3f8f26665a7"
format_stamp: "Formatted at 2026-08-11 02:40:24 on dist-test-slave-cvwc"
server_key: "e6c655fdc8ba14a26432930ee58a0905"
server_key_iv: "82c1997fffe1f2fed3aa6d016fc1c699"
server_key_version: "encryptionkey@0"
I20260811 02:40:24.882440 29270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "933ffd70c9a24a78a4bae3f8f26665a7"
format_stamp: "Formatted at 2026-08-11 02:40:24 on dist-test-slave-cvwc"
server_key: "e6c655fdc8ba14a26432930ee58a0905"
server_key_iv: "82c1997fffe1f2fed3aa6d016fc1c699"
server_key_version: "encryptionkey@0"
I20260811 02:40:24.883832 29270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:40:24.884740 29301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:24.884959 29270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:24.885046 29270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "933ffd70c9a24a78a4bae3f8f26665a7"
format_stamp: "Formatted at 2026-08-11 02:40:24 on dist-test-slave-cvwc"
server_key: "e6c655fdc8ba14a26432930ee58a0905"
server_key_iv: "82c1997fffe1f2fed3aa6d016fc1c699"
server_key_version: "encryptionkey@0"
I20260811 02:40:24.885109 29270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:24.900619 29270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:24.904029 29270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:24.904217 29270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:24.908656 29270 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:38957
I20260811 02:40:24.908692 29395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:38957 every 8 connection(s)
I20260811 02:40:24.909085 29270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:24.910153 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:24.912632 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap starting.
I20260811 02:40:24.913295 29396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:24.913589 29396 log.cc:824] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:24.914608 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 29270
I20260811 02:40:24.914700 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:40:24.914786 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: No bootstrap required, opened a new log
I20260811 02:40:24.914904 3052 external_mini_cluster.cc:1428] Setting key ccec7fd7e2903e884e18b924cfa0232f
I20260811 02:40:24.917230 29396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:24.917443 29396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:24.917480 29396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933ffd70c9a24a78a4bae3f8f26665a7, State: Initialized, Role: FOLLOWER
I20260811 02:40:24.917654 29396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:24.917800 29396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:40:24.917836 29396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:40:24.917899 29396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:24.919312 29396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:24.919451 29396 leader_election.cc:302] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933ffd70c9a24a78a4bae3f8f26665a7; no voters:
I20260811 02:40:24.920049 29396 leader_election.cc:288] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:40:24.920054 29402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:24.920499 29402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 LEADER]: Becoming Leader. State: Replica: 933ffd70c9a24a78a4bae3f8f26665a7, State: Running, Role: LEADER
I20260811 02:40:24.920663 29402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:24.920785 29396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:24.921111 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933ffd70c9a24a78a4bae3f8f26665a7. Latest consensus state: current_term: 1 leader_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } } }
I20260811 02:40:24.921384 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:24.921316 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } } }
I20260811 02:40:24.921669 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:24.922168 29409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:24.922691 29409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:40:24.924480 29409 catalog_manager.cc:1347] Generated new cluster ID: 633c06f5da2c403d9e20735f8fcf8b14
I20260811 02:40:24.924530 29409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:40:24 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:24.929653 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:24.915557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.2.251.62:38957)
0811 02:40:24.915758 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:24.915776 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:24.915810 (+ 34us) server_negotiation.cc:407] Connection header received
0811 02:40:24.917227 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:24.917238 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:24.917313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:24.917476 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:24.918650 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:24.919745 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:24.921556 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:24.922278 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:24.922410 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:24.925680 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:24.925685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:24.925686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:24.925700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:24.927817 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:24.928289 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:24.928292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:24.928293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:24.928424 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:24.928633 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:24.928633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:24.928633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:24.928962 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:24.929097 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:24.929268 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 02:40:24.929421 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260811 02:40:24.934844 29409 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:40:24.935604 29409 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:40:24.940514 29409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Generated new TSK 0
I20260811 02:40:24.940724 29409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:40:24.983057 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38957
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:25.076354 29441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:25.076581 29441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:25.076609 29441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:25.077837 29441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:25.077885 29441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:25.077912 29441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:40:25.079501 29441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:25.079733 29441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:25.079928 29441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:25.085736 29458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:25.085749 29452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:25.086017 29441 server_base.cc:1034] running on GCE node
W20260811 02:40:25.085749 29454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:25.086287 29441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:25.086557 29441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:25.087738 29441 hybrid_clock.cc:648] HybridClock initialized: now 1786416025087693 us; error 66 us; skew 500 ppm
Aug 11 02:40:25 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:25.090770 29441 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:40:25.091239 29441 webserver.cc:460] Webserver started at http://127.2.251.1:41415/ using document root <none> and password file <none>
I20260811 02:40:25.091483 29441 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:25.091535 29441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:25.091663 29441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:25.092829 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "f52a93b4ccf47a0683ff0cfecbd68f38"
server_key_iv: "b79abe09245e28afc82720a6cad5113b"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.093194 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "f52a93b4ccf47a0683ff0cfecbd68f38"
server_key_iv: "b79abe09245e28afc82720a6cad5113b"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.094697 29441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:40:25.095604 29475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.095901 29441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:25.096004 29441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "f52a93b4ccf47a0683ff0cfecbd68f38"
server_key_iv: "b79abe09245e28afc82720a6cad5113b"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.096076 29441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:25.119009 29441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:25.121976 29441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:25.122112 29441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:25.122323 29441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:25.122664 29441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:25.122700 29441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.122722 29441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:25.122733 29441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.128867 29441 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:40623
I20260811 02:40:25.128919 29661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:40623 every 8 connection(s)
I20260811 02:40:25.129261 29441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:40:25.130045 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 29441
I20260811 02:40:25.130172 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:40:25.130535 3052 external_mini_cluster.cc:1428] Setting key df00b99ee6de502ca9d526d4e1fca512
Aug 11 02:40:25 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:40:25.142656 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.131192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:51139 (local address 127.2.251.62:38957)
0811 02:40:25.131244 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.131246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.131517 (+ 271us) server_negotiation.cc:407] Connection header received
0811 02:40:25.131997 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.131999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.132012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.132088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:25.133086 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.134657 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.136317 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.136548 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.136616 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.139254 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:25.139255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:25.139255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:25.139268 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:25.140892 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.141443 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.141444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.141445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.141519 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.141918 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.141919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.141920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.142066 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:25.142133 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.142478 (+ 345us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.142574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:40:25.142783 29665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.131165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:51139)
0811 02:40:25.131361 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.131402 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.131900 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.132148 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.132152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.132765 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:25.132979 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.132984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.134880 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.134881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.136178 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.136183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.136640 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.136642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.136771 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.137307 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:25.137316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:25.139125 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:25.140992 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:25.141001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:25.141004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:25.141349 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:25.141626 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:25.141627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:25.141627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:25.141820 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:25.142153 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:25.142161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:25.142347 (+ 186us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.142441 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.142583 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260811 02:40:25.143649 29662 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38957
I20260811 02:40:25.143756 29662 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:25.143940 29662 heartbeater.cc:511] Master 127.2.251.62:38957 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
I20260811 02:40:25.144505 29328 ts_manager.cc:194] Registered new tserver with Master: 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623)
I20260811 02:40:25.145153 29328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:51139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:40:25.165606 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38957
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:25.252147 29670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:25.252586 29670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:25.252651 29670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:25.254166 29670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:25.254240 29670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:25.254288 29670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:40:25.256086 29670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:25.256398 29670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:25.256644 29670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:25.263342 29676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:25.263343 29677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:25.263386 29681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:25.263870 29670 server_base.cc:1034] running on GCE node
I20260811 02:40:25.264180 29670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:25.264524 29670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:25.265789 29670 hybrid_clock.cc:648] HybridClock initialized: now 1786416025265744 us; error 40 us; skew 500 ppm
Aug 11 02:40:25 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:25.268594 29670 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:40:25.269119 29670 webserver.cc:460] Webserver started at http://127.2.251.2:46835/ using document root <none> and password file <none>
I20260811 02:40:25.269361 29670 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:25.269418 29670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:25.269564 29670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:25.270819 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "c1aa4d8345f04d859c9c66213c0cbea5"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "cc1df7352651e0739199d6f1dece02b0"
server_key_iv: "793d386ad9af3ebb5f6e6ee4f04638a6"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.271214 29670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "c1aa4d8345f04d859c9c66213c0cbea5"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "cc1df7352651e0739199d6f1dece02b0"
server_key_iv: "793d386ad9af3ebb5f6e6ee4f04638a6"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.272759 29670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:40:25.273600 29693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.273828 29670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:25.273926 29670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "c1aa4d8345f04d859c9c66213c0cbea5"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "cc1df7352651e0739199d6f1dece02b0"
server_key_iv: "793d386ad9af3ebb5f6e6ee4f04638a6"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.274008 29670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:25.291173 29670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:25.293438 29670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:25.293591 29670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:25.293871 29670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:25.294214 29670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:25.294248 29670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.294277 29670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:25.294301 29670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.300612 29670 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38691
I20260811 02:40:25.300711 29881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38691 every 8 connection(s)
I20260811 02:40:25.301146 29670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:40:25.302529 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 29670
I20260811 02:40:25.302649 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:40:25.302845 3052 external_mini_cluster.cc:1428] Setting key e637dd1f0c7bca59bbb3fcdbf4e4289a
Aug 11 02:40:25 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:25.311707 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.302313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:48941 (local address 127.2.251.62:38957)
0811 02:40:25.302406 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.302409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.302574 (+ 165us) server_negotiation.cc:407] Connection header received
0811 02:40:25.303208 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.303209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.303219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.303263 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:25.304306 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.304824 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.306260 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.306470 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.306537 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.308822 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:25.308824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:25.308824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:25.308832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:25.310318 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.310735 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.310736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.310737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.310791 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.311037 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.311038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.311038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.311161 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:25.311218 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.311511 (+ 293us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.311608 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 02:40:25.311905 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.302240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:48941)
0811 02:40:25.302444 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.302488 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.303104 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.303350 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.303354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.303988 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:25.304196 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.304201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.304917 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.304918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.306141 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.306145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.306573 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.306574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.306693 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.307233 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:25.307250 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:25.308720 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:25.310412 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:25.310421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:25.310424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:25.310664 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:25.310850 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:25.310851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:25.310851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:25.310961 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:25.311217 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:25.311224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:25.311389 (+ 165us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.311486 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.311667 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
I20260811 02:40:25.312434 29882 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38957
I20260811 02:40:25.312563 29882 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:25.312777 29882 heartbeater.cc:511] Master 127.2.251.62:38957 requested a full tablet report, sending...
I20260811 02:40:25.313194 29328 ts_manager.cc:194] Registered new tserver with Master: c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
I20260811 02:40:25.313699 29328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:48941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:40:25.336966 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38957
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:25.433060 29894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:25.433323 29894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:25.433350 29894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:25.436934 29894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:25.437033 29894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:25.437073 29894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:40:25.441144 29894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:25.441627 29894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:25.441944 29894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:25.448541 29901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:25.448515 29907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:25.448498 29903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:25.448552 29894 server_base.cc:1034] running on GCE node
I20260811 02:40:25.448951 29894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:25.449213 29894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:25.450381 29894 hybrid_clock.cc:648] HybridClock initialized: now 1786416025450365 us; error 30 us; skew 500 ppm
Aug 11 02:40:25 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:25.453650 29894 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:40:25.454380 29894 webserver.cc:460] Webserver started at http://127.2.251.3:46345/ using document root <none> and password file <none>
I20260811 02:40:25.454633 29894 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:25.454695 29894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:25.454825 29894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:40:25.456058 29894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "920a3cb81d20d0f4a77061dd623ea468"
server_key_iv: "e2a070fb8287e50d55afd9b553110995"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.456418 29894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "920a3cb81d20d0f4a77061dd623ea468"
server_key_iv: "e2a070fb8287e50d55afd9b553110995"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.458096 29894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:40:25.459416 29921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.459802 29894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:25.459920 29894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "920a3cb81d20d0f4a77061dd623ea468"
server_key_iv: "e2a070fb8287e50d55afd9b553110995"
server_key_version: "encryptionkey@0"
I20260811 02:40:25.459998 29894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:25.477052 29894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:25.480204 29894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:25.480374 29894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:25.480640 29894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:25.480991 29894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:25.481027 29894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.481058 29894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:25.481083 29894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:25.487217 29894 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:35273
I20260811 02:40:25.487376 30117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:35273 every 8 connection(s)
I20260811 02:40:25.487659 29894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
Aug 11 02:40:25 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:25.493922 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 29894
I20260811 02:40:25.494004 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:40:25.494202 3052 external_mini_cluster.cc:1428] Setting key b8201692370afade8d5a4bf748148e42
I20260811 02:40:25.498265 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.488412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:46557 (local address 127.2.251.62:38957)
0811 02:40:25.488463 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.488466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.488680 (+ 214us) server_negotiation.cc:407] Connection header received
0811 02:40:25.489220 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.489220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.489233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.489287 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:25.490331 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.490841 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.491833 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.492038 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.492095 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.494586 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:25.494590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:25.494590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:25.494604 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:25.496355 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.496829 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.496830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.496831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.496901 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.497460 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.497461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.497461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.497595 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:25.497663 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.498109 (+ 446us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.498194 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:40:25.498711 30121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.488400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:46557)
0811 02:40:25.488552 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.488591 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.489107 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.489358 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.489362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.489993 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:25.490225 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.490230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.490929 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.490931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.491741 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.491744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.492112 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.492113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.492242 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.492751 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:25.492760 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:25.494429 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:25.496459 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:25.496466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:25.496468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:25.496727 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:25.497036 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:25.497038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:25.497038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:25.497352 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:25.497701 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:25.497708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:25.497937 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.498047 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.498208 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260811 02:40:25.499414 30118 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38957
I20260811 02:40:25.499527 30118 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:25.499718 30118 heartbeater.cc:511] Master 127.2.251.62:38957 requested a full tablet report, sending...
I20260811 02:40:25.500293 29328 ts_manager.cc:194] Registered new tserver with Master: fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273)
I20260811 02:40:25.500692 29328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:46557
I20260811 02:40:25.507066 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:40:25.515260 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.508526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.2.251.62:38957)
0811 02:40:25.508609 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.508611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.508685 (+ 74us) server_negotiation.cc:407] Connection header received
0811 02:40:25.508756 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.508757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.508769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.508824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:25.509410 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.510067 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.510864 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.511067 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.511124 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.512598 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:25.512600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:25.512600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:25.512610 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:25.514125 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.514474 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.514475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.514475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.514532 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:25.514693 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:25.514694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:25.514694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:25.514862 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:25.514958 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.515074 (+ 116us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.515163 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 02:40:25.517486 29328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:40:25.519299 29328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:40:25.526852 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.522161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:35273 (local address 127.0.0.1:60988)
0811 02:40:25.522486 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.522496 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.522569 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.522937 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.522941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.522943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:25.523037 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.523039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.524380 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.524381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.525411 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.525413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.526493 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.526494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.526585 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.526603 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.526682 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.526752 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260811 02:40:25.526878 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.522583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.2.251.1:40623)
0811 02:40:25.522852 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.522856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.522884 (+ 28us) server_negotiation.cc:407] Connection header received
0811 02:40:25.522994 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.522999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.523017 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.523090 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:25.523271 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.523963 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.525417 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.526370 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.526455 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.526525 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.526738 (+ 213us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.526783 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20260811 02:40:25.526877 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.522315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.2.251.3:35273)
0811 02:40:25.522762 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.522766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.522774 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:40:25.522813 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.522815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.522830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.522923 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:25.523115 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.524264 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.525505 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.526381 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.526487 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.526538 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.526708 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.526773 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":89,"threads_started":1}
I20260811 02:40:25.527182 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.522339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40623 (local address 127.0.0.1:46288)
0811 02:40:25.522568 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.522577 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.522616 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.523162 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.523163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.523164 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:25.523203 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.523204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.524058 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.524059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.525313 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.525316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.526562 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.526563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.526640 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.526645 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.526692 (+ 47us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.527118 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":371,"thread_start_us":79,"threads_started":1}
I20260811 02:40:25.527699 30003 tablet_service.cc:1467] Processing CreateTablet for tablet 2295388778df4239b93403ca8bc075d4 (DEFAULT_TABLE table=test-table [id=28a032448ef147928c980b38f23c8c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:25.528082 30003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2295388778df4239b93403ca8bc075d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:25.528085 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.522127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38691 (local address 127.0.0.1:52686)
0811 02:40:25.522447 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.522490 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.522567 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.522955 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.522956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.522958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:25.523024 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.523026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.525493 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.525493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.526330 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.526333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.527859 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.527860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.527928 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.527932 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.527974 (+ 42us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.528017 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":58,"threads_started":1}
I20260811 02:40:25.528287 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.522249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.2.251.2:38691)
0811 02:40:25.522460 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.522465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.522650 (+ 185us) server_negotiation.cc:407] Connection header received
0811 02:40:25.522700 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.522703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.522719 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.522805 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:25.523106 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.524284 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.526434 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.527769 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.527843 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.527962 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.527993 (+ 31us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.528034 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20260811 02:40:25.528877 29546 tablet_service.cc:1467] Processing CreateTablet for tablet 2295388778df4239b93403ca8bc075d4 (DEFAULT_TABLE table=test-table [id=28a032448ef147928c980b38f23c8c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:25.528910 29774 tablet_service.cc:1467] Processing CreateTablet for tablet 2295388778df4239b93403ca8bc075d4 (DEFAULT_TABLE table=test-table [id=28a032448ef147928c980b38f23c8c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:25.529264 29774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2295388778df4239b93403ca8bc075d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:25.529264 29546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2295388778df4239b93403ca8bc075d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:25.531227 30150 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Bootstrap starting.
I20260811 02:40:25.531955 30150 tablet_bootstrap.cc:654] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:25.532482 30150 log.cc:824] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:25.532963 30152 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap starting.
I20260811 02:40:25.533303 30150 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: No bootstrap required, opened a new log
I20260811 02:40:25.533382 30150 ts_tablet_manager.cc:1397] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:40:25.533579 30152 tablet_bootstrap.cc:654] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:25.534544 30152 log.cc:824] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:25.535400 30150 raft_consensus.cc:348] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.535585 30150 raft_consensus.cc:374] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:25.535621 30150 raft_consensus.cc:729] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff2de8f1d074bcab5c7bea1e0cb1fea, State: Initialized, Role: FOLLOWER
I20260811 02:40:25.535826 30150 consensus_queue.cc:260] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.535998 30156 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Bootstrap starting.
I20260811 02:40:25.536242 30150 ts_tablet_manager.cc:1428] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:40:25.536391 30118 heartbeater.cc:503] Master 127.2.251.62:38957 was elected leader, sending a full tablet report...
I20260811 02:40:25.537315 30156 tablet_bootstrap.cc:654] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:25.538059 30156 log.cc:824] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:25.539136 30152 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: No bootstrap required, opened a new log
I20260811 02:40:25.539170 30156 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: No bootstrap required, opened a new log
I20260811 02:40:25.539381 30156 ts_tablet_manager.cc:1397] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260811 02:40:25.539673 30152 ts_tablet_manager.cc:1397] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260811 02:40:25.541440 30156 raft_consensus.cc:348] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.541689 30156 raft_consensus.cc:374] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:25.541780 30156 raft_consensus.cc:729] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1aa4d8345f04d859c9c66213c0cbea5, State: Initialized, Role: FOLLOWER
I20260811 02:40:25.541960 30156 consensus_queue.cc:260] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.542328 30156 ts_tablet_manager.cc:1428] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 02:40:25.542539 29882 heartbeater.cc:503] Master 127.2.251.62:38957 was elected leader, sending a full tablet report...
I20260811 02:40:25.543433 30152 raft_consensus.cc:348] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.543715 30152 raft_consensus.cc:374] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:25.543772 30152 raft_consensus.cc:729] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0a47ed487f4e0592201cb68d2c4a3c, State: Initialized, Role: FOLLOWER
I20260811 02:40:25.543936 30152 consensus_queue.cc:260] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.544411 30152 ts_tablet_manager.cc:1428] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260811 02:40:25.544492 29662 heartbeater.cc:503] Master 127.2.251.62:38957 was elected leader, sending a full tablet report...
W20260811 02:40:25.552043 29883 tablet.cc:2369] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:40:25.582099 30163 raft_consensus.cc:482] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:25.582297 30163 raft_consensus.cc:504] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.582762 30163 leader_election.cc:288] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623), fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273)
I20260811 02:40:25.586258 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.583203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:52711 (local address 127.2.251.1:40623)
0811 02:40:25.583290 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.583293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.583509 (+ 216us) server_negotiation.cc:407] Connection header received
0811 02:40:25.583547 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.583548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.583564 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.583615 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:25.583788 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.584293 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.585443 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.585900 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.585975 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.585989 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.586131 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.586172 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 02:40:25.586300 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.583225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:41723 (local address 127.2.251.3:35273)
0811 02:40:25.583295 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.583298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.583310 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:40:25.583345 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.583346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.583359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.583428 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:25.583671 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.584335 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.585448 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.585935 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.586014 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.586028 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.586178 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.586225 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:40:25.586308 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.583045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:35273 (local address 127.2.251.2:41723)
0811 02:40:25.583110 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.583122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.583174 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.583454 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.583456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.583459 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:25.583585 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.583588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.584488 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.584489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.585328 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.585330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.586021 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.586023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.586111 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.586114 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.586163 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.586210 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260811 02:40:25.586309 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.583049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40623 (local address 127.2.251.2:52711)
0811 02:40:25.583421 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.583431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:25.583498 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:25.583626 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:25.583628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:25.583629 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:25.583722 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.583725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.584379 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.584380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.585337 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:25.585341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.585980 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.585981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:25.586058 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.586063 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:40:25.586112 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 02:40:25.586165 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":209,"thread_start_us":147,"threads_started":1}
I20260811 02:40:25.586799 30038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2295388778df4239b93403ca8bc075d4" candidate_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" is_pre_election: true
I20260811 02:40:25.586794 29584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2295388778df4239b93403ca8bc075d4" candidate_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" is_pre_election: true
I20260811 02:40:25.586980 30038 raft_consensus.cc:2454] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1aa4d8345f04d859c9c66213c0cbea5 in term 0.
I20260811 02:40:25.586982 29584 raft_consensus.cc:2454] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1aa4d8345f04d859c9c66213c0cbea5 in term 0.
I20260811 02:40:25.587277 29694 leader_election.cc:302] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0a47ed487f4e0592201cb68d2c4a3c, c1aa4d8345f04d859c9c66213c0cbea5; no voters:
I20260811 02:40:25.587456 30163 raft_consensus.cc:2781] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:40:25.587539 30163 raft_consensus.cc:482] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:40:25.587585 30163 raft_consensus.cc:3037] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:25.588431 30163 raft_consensus.cc:504] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.588581 30163 leader_election.cc:288] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [CANDIDATE]: Term 1 election: Requested vote from peers 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623), fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273)
I20260811 02:40:25.588835 29584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2295388778df4239b93403ca8bc075d4" candidate_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
I20260811 02:40:25.588824 30038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2295388778df4239b93403ca8bc075d4" candidate_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea"
I20260811 02:40:25.588917 29584 raft_consensus.cc:3037] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:25.588917 30038 raft_consensus.cc:3037] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:25.589888 29584 raft_consensus.cc:2454] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1aa4d8345f04d859c9c66213c0cbea5 in term 1.
I20260811 02:40:25.589888 30038 raft_consensus.cc:2454] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1aa4d8345f04d859c9c66213c0cbea5 in term 1.
I20260811 02:40:25.590114 29696 leader_election.cc:302] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea; no voters:
I20260811 02:40:25.590241 30163 raft_consensus.cc:2781] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:25.590545 30163 raft_consensus.cc:686] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 LEADER]: Becoming Leader. State: Replica: c1aa4d8345f04d859c9c66213c0cbea5, State: Running, Role: LEADER
I20260811 02:40:25.590751 30163 consensus_queue.cc:237] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:25.592002 29328 catalog_manager.cc:5495] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2). New cstate: current_term: 1 leader_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } health_report { overall_health: UNKNOWN } } }
I20260811 02:40:25.603549 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.600430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.2.251.2:38691)
0811 02:40:25.600537 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:25.600539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:25.600547 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:40:25.600599 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:25.600599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:25.600628 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:25.600702 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:40:25.600964 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.601480 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.602470 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:25.602750 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:25.602819 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:25.603121 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:40:25.603250 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:40:25.603328 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:25.603400 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 02:40:25.603424 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 02:40:25.605691 30038 raft_consensus.cc:1261] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Refusing update from remote peer c1aa4d8345f04d859c9c66213c0cbea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:25.605691 29584 raft_consensus.cc:1261] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Refusing update from remote peer c1aa4d8345f04d859c9c66213c0cbea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:25.606141 30163 consensus_queue.cc:1035] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:25.606293 30168 consensus_queue.cc:1035] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:25.609047 30174 mvcc.cc:205] Tried to move back new op lower bound from 7317160040877854720 to 7317160040820424704. Current Snapshot: MvccSnapshot[applied={T|T < 7317160040877854720}]
I20260811 02:40:25.610307 30175 mvcc.cc:205] Tried to move back new op lower bound from 7317160040877854720 to 7317160040820424704. Current Snapshot: MvccSnapshot[applied={T|T < 7317160040877854720 or (T in {7317160040877854720})}]
I20260811 02:40:25.612215 29327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:25.612401 29327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:25.613606 29327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Sending DeleteTablet for 3 replicas of tablet 2295388778df4239b93403ca8bc075d4
I20260811 02:40:25.614183 29774 tablet_service.cc:1514] Processing DeleteTablet for tablet 2295388778df4239b93403ca8bc075d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:25 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:52686
I20260811 02:40:25.614229 29546 tablet_service.cc:1514] Processing DeleteTablet for tablet 2295388778df4239b93403ca8bc075d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:25 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:46288
I20260811 02:40:25.614226 30003 tablet_service.cc:1514] Processing DeleteTablet for tablet 2295388778df4239b93403ca8bc075d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:25 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:60988
I20260811 02:40:25.614570 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 29441
I20260811 02:40:25.628039 30196 tablet_replica.cc:330] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: stopping tablet replica
I20260811 02:40:25.628266 30196 raft_consensus.cc:2229] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:40:25.628489 30196 raft_consensus.cc:2258] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:25.631497 30193 tablet_replica.cc:330] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: stopping tablet replica
I20260811 02:40:25.631707 30196 ts_tablet_manager.cc:1905] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:25.631798 30193 raft_consensus.cc:2229] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:40:25.631945 30193 raft_consensus.cc:2258] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:25.633846 30196 ts_tablet_manager.cc:1918] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:25.633934 30196 log.cc:1196] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/2295388778df4239b93403ca8bc075d4
I20260811 02:40:25.634256 30193 ts_tablet_manager.cc:1905] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:25.635718 30193 ts_tablet_manager.cc:1918] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:25.635743 30196 ts_tablet_manager.cc:1939] T 2295388778df4239b93403ca8bc075d4 P c1aa4d8345f04d859c9c66213c0cbea5: Deleting consensus metadata
I20260811 02:40:25.635893 30193 log.cc:1196] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/2295388778df4239b93403ca8bc075d4
I20260811 02:40:25.636270 29302 catalog_manager.cc:4841] TS c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): tablet 2295388778df4239b93403ca8bc075d4 (table test-table [id=28a032448ef147928c980b38f23c8c2c]) successfully deleted
I20260811 02:40:25.636400 30193 ts_tablet_manager.cc:1939] T 2295388778df4239b93403ca8bc075d4 P fff2de8f1d074bcab5c7bea1e0cb1fea: Deleting consensus metadata
I20260811 02:40:25.636812 29306 catalog_manager.cc:4841] TS fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): tablet 2295388778df4239b93403ca8bc075d4 (table test-table [id=28a032448ef147928c980b38f23c8c2c]) successfully deleted
W20260811 02:40:25.639387 29302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:40623 (error 108)
I20260811 02:40:25.639506 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 29670
I20260811 02:40:25.650087 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:25.649795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40623 (local address 127.0.0.1:46302)
0811 02:40:25.649960 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:25.650031 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260811 02:40:25.650266 29302 catalog_manager.cc:4584] TS 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2295388778df4239b93403ca8bc075d4: Network error: Client connection negotiation failed: client connection to 127.2.251.1:40623: connect: Connection refused (error 111)
I20260811 02:40:25.663009 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 29894
I20260811 02:40:25.678637 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 29270
I20260811 02:40:45.697364 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38957
--webserver_interface=127.2.251.62
--webserver_port=46731
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:40:45.788707 30198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:45.788971 30198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:45.788995 30198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:45.790290 30198 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 02:40:45.790339 30198 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 02:40:45.790352 30198 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 02:40:45.790362 30198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:40:45.790374 30198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:45.790386 30198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:45.790397 30198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:40:45.790410 30198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:40:45.792024 30198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=46731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:45.792286 30198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:45.792541 30198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:40:45.804657 30198 server_base.cc:1034] running on GCE node
W20260811 02:40:45.804621 30211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:45.804638 30206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:45.804622 30207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:45.805235 30198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:45.805670 30198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:45.806908 30198 hybrid_clock.cc:648] HybridClock initialized: now 1786416045806881 us; error 48 us; skew 500 ppm
Aug 11 02:40:45 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:45.814858 30198 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:40:45.815433 30198 webserver.cc:460] Webserver started at http://127.2.251.62:46731/ using document root <none> and password file <none>
I20260811 02:40:45.815678 30198 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:45.815759 30198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:45.817500 30198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:40:45.818749 30218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:45.819113 30198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:40:45.819258 30198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "933ffd70c9a24a78a4bae3f8f26665a7"
format_stamp: "Formatted at 2026-08-11 02:40:24 on dist-test-slave-cvwc"
server_key: "e6c655fdc8ba14a26432930ee58a0905"
server_key_iv: "82c1997fffe1f2fed3aa6d016fc1c699"
server_key_version: "encryptionkey@0"
I20260811 02:40:45.820035 30198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:45.836719 30198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:45.840879 30198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:45.841111 30198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:45.846855 30198 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:38957
I20260811 02:40:45.846939 30317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:38957 every 8 connection(s)
I20260811 02:40:45.847436 30198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:40:45.849323 30319 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:40:45.850200 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap starting.
I20260811 02:40:45.854928 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 30198
I20260811 02:40:45.855301 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:40623
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:38957
--webserver_port=41415
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:45.859786 30319 log.cc:824] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:45.861968 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:40:45.862291 30319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap complete.
I20260811 02:40:45.864545 30319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:45.864794 30319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933ffd70c9a24a78a4bae3f8f26665a7, State: Initialized, Role: FOLLOWER
I20260811 02:40:45.864950 30319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:45.865021 30319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:40:45.865072 30319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:40:45.865128 30319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 1 FOLLOWER]: Advancing to term 2
I20260811 02:40:45.866470 30319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:45.866576 30319 leader_election.cc:302] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933ffd70c9a24a78a4bae3f8f26665a7; no voters:
I20260811 02:40:45.866746 30319 leader_election.cc:288] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 02:40:45.866808 30329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 02:40:45.866945 30329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 LEADER]: Becoming Leader. State: Replica: 933ffd70c9a24a78a4bae3f8f26665a7, State: Running, Role: LEADER
I20260811 02:40:45.867050 30319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:40:45.867105 30329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:40:45.867468 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933ffd70c9a24a78a4bae3f8f26665a7. Latest consensus state: current_term: 2 leader_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } } }
I20260811 02:40:45.867555 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:45.867697 30330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } } }
I20260811 02:40:45.867775 30330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:40:45.868110 30339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:40:45.869009 30339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:40:45.869119 30339 catalog_manager.cc:1259] Loaded cluster ID: 633c06f5da2c403d9e20735f8fcf8b14
I20260811 02:40:45.869196 30339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:40:45.870230 30339 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:40:45.870460 30339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Loaded TSK: 0
I20260811 02:40:45.890169 30339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Generated new TSK 1
I20260811 02:40:45.890340 30339 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 02:40:45.973204 30325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:45.973453 30325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:45.973496 30325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:45.974872 30325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:45.974922 30325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:45.974963 30325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:40:45.976586 30325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:40623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=41415
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:45.976853 30325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:45.977087 30325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:45.983342 30367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:45.983347 30372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:45.983379 30369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:45.984047 30325 server_base.cc:1034] running on GCE node
I20260811 02:40:45.984295 30325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:45.984562 30325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:45.985767 30325 hybrid_clock.cc:648] HybridClock initialized: now 1786416045985749 us; error 35 us; skew 500 ppm
Aug 11 02:40:45 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:45.988562 30325 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:40:45.989209 30325 webserver.cc:460] Webserver started at http://127.2.251.1:41415/ using document root <none> and password file <none>
I20260811 02:40:45.989473 30325 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:45.989559 30325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:45.991520 30325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:45.992422 30384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:45.992627 30325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:45.992729 30325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "f52a93b4ccf47a0683ff0cfecbd68f38"
server_key_iv: "b79abe09245e28afc82720a6cad5113b"
server_key_version: "encryptionkey@0"
I20260811 02:40:45.993243 30325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:46.022792 30325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:46.026224 30325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:46.026552 30325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:46.026901 30325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:46.027400 30393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 02:40:46.029486 30325 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 02:40:46.029560 30325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 02:40:46.029654 30325 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 02:40:46.030701 30325 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 02:40:46.030751 30325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:46.030900 30393 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap starting.
I20260811 02:40:46.038430 30325 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:40623
I20260811 02:40:46.039101 30325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:40:46.040239 30572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:40623 every 8 connection(s)
I20260811 02:40:46.043604 30393 log.cc:824] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:46.044080 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 30325
I20260811 02:40:46.044404 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:38691
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:38957
--webserver_port=46835
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:46.046581 30393 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:40:46.046911 30393 tablet_bootstrap.cc:492] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap complete.
I20260811 02:40:46.047948 30393 ts_tablet_manager.cc:1397] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
Aug 11 02:40:46 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:46.049237 30393 raft_consensus.cc:348] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:46.049500 30393 raft_consensus.cc:729] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0a47ed487f4e0592201cb68d2c4a3c, State: Initialized, Role: FOLLOWER
I20260811 02:40:46.049659 30393 consensus_queue.cc:260] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } } peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } }
I20260811 02:40:46.050038 30393 ts_tablet_manager.cc:1428] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260811 02:40:46.053683 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.041089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:34299 (local address 127.2.251.62:38957)
0811 02:40:46.041860 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.041872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.041933 (+ 61us) server_negotiation.cc:407] Connection header received
0811 02:40:46.042717 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.042724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.042782 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.042918 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:46.043692 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.044997 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.046571 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.047169 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.047284 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.050118 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:46.050124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:46.050124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:46.050139 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:46.051880 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.052335 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.052338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.052339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.052441 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.052690 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.052691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.052691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.052944 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:46.053022 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.053345 (+ 323us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.053482 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":113,"threads_started":1}
I20260811 02:40:46.053684 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.040964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:34299)
0811 02:40:46.041353 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.041405 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.042368 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.042974 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.042978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.043337 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:46.043558 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.043565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.045165 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.045166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.046455 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.046459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.047296 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.047297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.047448 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.048153 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:46.048164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:46.049995 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:46.051974 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.051983 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.051986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.052233 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.052505 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.052506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.052506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.052620 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.053014 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:46.053021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:46.053165 (+ 144us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.053254 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.053461 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":152,"threads_started":1}
I20260811 02:40:46.054606 30573 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38957
I20260811 02:40:46.054731 30573 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:46.054956 30573 heartbeater.cc:511] Master 127.2.251.62:38957 requested a full tablet report, sending...
I20260811 02:40:46.056254 30248 ts_manager.cc:194] Registered new tserver with Master: 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623)
I20260811 02:40:46.057755 30248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:34299
I20260811 02:40:46.059090 30573 heartbeater.cc:503] Master 127.2.251.62:38957 was elected leader, sending a full tablet report...
Aug 11 02:40:46 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.1@KRBTEST.COM
I20260811 02:40:46.066632 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.057186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40623 (local address 127.0.0.1:44408)
0811 02:40:46.057487 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.057529 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.057610 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.058022 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.058029 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.058475 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:46.058550 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.058554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.059515 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.059516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.060271 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.060275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.060748 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.060749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.060837 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.061352 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:46.061358 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:46.062980 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:46.064793 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.064804 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.064806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.065098 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.065388 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.065388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.065389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.065463 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.066131 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:46.066139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:46.066222 (+ 83us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.066339 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.066512 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260811 02:40:46.066700 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.057295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.2.251.1:40623)
0811 02:40:46.057815 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.057820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.057831 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:40:46.057871 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.057874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.057891 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.058006 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:46.058664 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.059438 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.060348 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.060672 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.060738 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.063108 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:46.063119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:46.063119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:46.063128 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:46.064692 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.065228 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.065233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.065233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.065311 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.065622 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.065625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.065626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.066025 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:46.066152 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.066444 (+ 292us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.066602 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":157,"threads_started":1}
I20260811 02:40:46.067248 30464 tablet_service.cc:1514] Processing DeleteTablet for tablet 2295388778df4239b93403ca8bc075d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:25 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:44408
I20260811 02:40:46.068171 30587 tablet_replica.cc:330] stopping tablet replica
I20260811 02:40:46.068321 30587 raft_consensus.cc:2229] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:40:46.068409 30587 raft_consensus.cc:2258] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:46.068828 30587 ts_tablet_manager.cc:1905] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:46.071266 30587 ts_tablet_manager.cc:1918] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:46.071348 30587 log.cc:1196] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal/wals/2295388778df4239b93403ca8bc075d4
I20260811 02:40:46.071573 30587 ts_tablet_manager.cc:1939] T 2295388778df4239b93403ca8bc075d4 P 4e0a47ed487f4e0592201cb68d2c4a3c: Deleting consensus metadata
I20260811 02:40:46.072290 30223 catalog_manager.cc:4841] TS 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623): tablet 2295388778df4239b93403ca8bc075d4 (table test-table [id=28a032448ef147928c980b38f23c8c2c]) successfully deleted
W20260811 02:40:46.142932 30579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:46.143190 30579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:46.143236 30579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:46.144570 30579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:46.144629 30579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:46.144671 30579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:40:46.146692 30579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:38691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=46835
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:46.147033 30579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:46.147300 30579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:46.153107 30597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:46.153107 30601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:46.153146 30596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:46.153424 30579 server_base.cc:1034] running on GCE node
I20260811 02:40:46.153795 30579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:46.154090 30579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:46.155337 30579 hybrid_clock.cc:648] HybridClock initialized: now 1786416046155321 us; error 33 us; skew 500 ppm
Aug 11 02:40:46 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:46.158221 30579 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:40:46.158766 30579 webserver.cc:460] Webserver started at http://127.2.251.2:46835/ using document root <none> and password file <none>
I20260811 02:40:46.159016 30579 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:46.159088 30579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:46.160771 30579 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:46.161533 30611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:46.161759 30579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:46.161859 30579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "c1aa4d8345f04d859c9c66213c0cbea5"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "cc1df7352651e0739199d6f1dece02b0"
server_key_iv: "793d386ad9af3ebb5f6e6ee4f04638a6"
server_key_version: "encryptionkey@0"
I20260811 02:40:46.162231 30579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:46.172640 30579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:46.176072 30579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:46.176255 30579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:46.176524 30579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:46.176867 30579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:46.176904 30579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:46.176935 30579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:46.176954 30579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:46.183708 30579 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38691
I20260811 02:40:46.183866 30796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38691 every 8 connection(s)
I20260811 02:40:46.184182 30579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
Aug 11 02:40:46 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:46.194207 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 30579
I20260811 02:40:46.194603 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:35273
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:38957
--webserver_port=46345
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:40:46.196199 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.185032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:46175 (local address 127.2.251.62:38957)
0811 02:40:46.185119 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.185121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.185284 (+ 163us) server_negotiation.cc:407] Connection header received
0811 02:40:46.185841 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.185842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.185856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.185921 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:46.187083 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.187821 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.188921 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.189159 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.189228 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.192490 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:46.192492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:46.192492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:46.192501 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:46.194345 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.195066 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.195068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.195069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.195148 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.195454 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.195455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.195455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.195590 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:46.195648 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.196009 (+ 361us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.196122 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 02:40:46.196413 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.184994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:46175)
0811 02:40:46.185161 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.185203 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.185735 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.185997 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.186000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.186686 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:46.186897 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.186903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.187961 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.187963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.188816 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.188819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.189323 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.189325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.189587 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.190397 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:46.190410 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:46.192369 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:46.194552 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.194565 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.194569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.194976 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.195227 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.195227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.195228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.195383 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.195651 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:46.195659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:46.195843 (+ 184us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.195971 (+ 128us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.196167 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260811 02:40:46.196960 30797 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38957
I20260811 02:40:46.197104 30797 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:46.197327 30797 heartbeater.cc:511] Master 127.2.251.62:38957 requested a full tablet report, sending...
I20260811 02:40:46.197854 30248 ts_manager.cc:194] Registered new tserver with Master: c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
I20260811 02:40:46.198426 30248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:46175
W20260811 02:40:46.289256 30802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:40:46.289500 30802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:40:46.289535 30802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:40:46.290804 30802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:40:46.290866 30802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:40:46.290905 30802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:40:46.292529 30802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:35273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=46345
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:40:46.292793 30802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:40:46.293036 30802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:40:46.298730 30809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:46.298838 30802 server_base.cc:1034] running on GCE node
W20260811 02:40:46.298707 30813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:40:46.298694 30810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:40:46.299252 30802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:40:46.299539 30802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:40:46.300721 30802 hybrid_clock.cc:648] HybridClock initialized: now 1786416046300703 us; error 43 us; skew 500 ppm
Aug 11 02:40:46 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:40:46.303574 30802 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:40:46.304100 30802 webserver.cc:460] Webserver started at http://127.2.251.3:46345/ using document root <none> and password file <none>
I20260811 02:40:46.304323 30802 fs_manager.cc:359] Metadata directory not provided
I20260811 02:40:46.304394 30802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:40:46.305965 30802 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:46.306862 30828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:46.307111 30802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:40:46.307214 30802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "920a3cb81d20d0f4a77061dd623ea468"
server_key_iv: "e2a070fb8287e50d55afd9b553110995"
server_key_version: "encryptionkey@0"
I20260811 02:40:46.307636 30802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:40:46.319509 30802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:40:46.322413 30802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:40:46.322582 30802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:40:46.322857 30802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:40:46.323186 30802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:40:46.323222 30802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:46.323253 30802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:40:46.323271 30802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:40:46.329888 30802 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:35273
I20260811 02:40:46.329952 31010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:35273 every 8 connection(s)
I20260811 02:40:46.330317 30802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:40:46.333006 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 30802
Aug 11 02:40:46 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:40:46.340361 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.333500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.2.251.62:38957)
0811 02:40:46.333701 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.333704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.333746 (+ 42us) server_negotiation.cc:407] Connection header received
0811 02:40:46.333799 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.333800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.333813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.333887 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:46.334598 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.335067 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.335886 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.336267 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.336350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.337761 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:46.337763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:46.337763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:46.337771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:46.339203 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.339598 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.339599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.339599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.339655 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.339824 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.339825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.339825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.339953 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:46.340127 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.340165 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.340249 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
I20260811 02:40:46.342036 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.331130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:56851 (local address 127.2.251.62:38957)
0811 02:40:46.331209 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.331211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.331483 (+ 272us) server_negotiation.cc:407] Connection header received
0811 02:40:46.332110 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.332113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.332133 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.332230 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:46.333549 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.334349 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.335822 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.336019 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.336075 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.338799 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:46.338800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:46.338800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:46.338808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:46.340132 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.340560 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.340562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.340562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.340615 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.340930 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.340934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.340934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.341181 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:46.341267 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.341830 (+ 563us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.341944 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 02:40:46.342255 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.331103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:56851)
0811 02:40:46.331298 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.331346 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.331895 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.332303 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.332307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.333159 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:40:46.333418 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.333424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.334486 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.334487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.335669 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.335679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.336159 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.336161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.336355 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.336936 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:40:46.336947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:40:46.338683 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:40:46.340214 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.340224 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.340226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.340477 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.340680 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:40:46.340682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:40:46.340682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:40:46.340832 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:40:46.341277 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:40:46.341295 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:40:46.341605 (+ 310us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.341791 (+ 186us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.342005 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260811 02:40:46.342882 31012 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38957
I20260811 02:40:46.342983 31012 heartbeater.cc:464] Registering TS with master...
I20260811 02:40:46.343156 31012 heartbeater.cc:511] Master 127.2.251.62:38957 requested a full tablet report, sending...
I20260811 02:40:46.343526 30248 ts_manager.cc:194] Registered new tserver with Master: fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273)
I20260811 02:40:46.343869 30248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:56851
I20260811 02:40:46.352850 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:40:46.356683 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.353365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.2.251.62:38957)
0811 02:40:46.353441 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.353442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.353697 (+ 255us) server_negotiation.cc:407] Connection header received
0811 02:40:46.353795 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.353796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.353842 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.353920 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:40:46.354235 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.354749 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.355696 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.356059 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.356126 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.356316 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:40:46.356380 (+ 64us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 02:40:46.356525 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46980: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 02:40:46.356793 30577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46980: token expired
W20260811 02:40:46.357641 30127 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.251.62:38957)
I20260811 02:40:46.364024 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.357997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.2.251.62:38957)
0811 02:40:46.358091 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.358093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.358103 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:40:46.358176 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.358177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.358186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.358231 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:40:46.358826 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.359313 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.360065 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.360248 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.360314 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.361594 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:40:46.361595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:40:46.361595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:40:46.361602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:40:46.363018 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.363400 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.363401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.363401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.363467 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:40:46.363617 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:40:46.363617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:40:46.363618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:40:46.363729 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:40:46.363820 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.363879 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.363950 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 02:40:46.365739 30248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:40:46.367569 30248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:40:46.370965 30464 tablet_service.cc:1467] Processing CreateTablet for tablet 51bca1a9896e4556b95376caaf5070c9 (DEFAULT_TABLE table=test-table [id=c6ad08d8ea684b1ebdd31505ec0bdbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:46.371217 30464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bca1a9896e4556b95376caaf5070c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:46.372860 30393 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap starting.
I20260811 02:40:46.373201 30393 tablet_bootstrap.cc:654] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:46.373672 30393 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: No bootstrap required, opened a new log
I20260811 02:40:46.373708 30393 ts_tablet_manager.cc:1397] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260811 02:40:46.374120 30393 raft_consensus.cc:348] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.374223 30393 raft_consensus.cc:374] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:46.374281 30393 raft_consensus.cc:729] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0a47ed487f4e0592201cb68d2c4a3c, State: Initialized, Role: FOLLOWER
I20260811 02:40:46.374339 30393 consensus_queue.cc:260] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.374432 30393 ts_tablet_manager.cc:1428] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 02:40:46.375486 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.370830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.2.251.3:35273)
0811 02:40:46.371028 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.371034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.371044 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:40:46.371086 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.371088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.371107 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.371368 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:46.371453 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.372150 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.374048 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.375070 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.375177 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.375250 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.375335 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.375391 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":111,"threads_started":1}
I20260811 02:40:46.375603 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.370596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:35273 (local address 127.0.0.1:49244)
0811 02:40:46.370748 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.370758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.370834 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.371218 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.371219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.371222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:46.371295 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.371297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.372235 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.372236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.373864 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.373871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.375178 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.375179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.375247 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.375265 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.375309 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.375518 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":21,"thread_start_us":68,"threads_started":1}
I20260811 02:40:46.375705 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.370590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38691 (local address 127.0.0.1:34264)
0811 02:40:46.370641 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.370654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.370693 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.371242 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.371243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.371245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:46.371297 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.371299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.372555 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.372557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.373871 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.373873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.375420 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.375421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.375525 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.375531 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.375598 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.375652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 02:40:46.376006 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.370849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.2.251.2:38691)
0811 02:40:46.371061 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.371065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.371076 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:40:46.371118 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.371120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.371134 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.371371 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:46.371456 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.372443 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.374144 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.375306 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.375389 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.375457 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.375866 (+ 409us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.375927 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":104,"threads_started":1}
I20260811 02:40:46.376631 30692 tablet_service.cc:1467] Processing CreateTablet for tablet 51bca1a9896e4556b95376caaf5070c9 (DEFAULT_TABLE table=test-table [id=c6ad08d8ea684b1ebdd31505ec0bdbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:46.376642 30903 tablet_service.cc:1467] Processing CreateTablet for tablet 51bca1a9896e4556b95376caaf5070c9 (DEFAULT_TABLE table=test-table [id=c6ad08d8ea684b1ebdd31505ec0bdbe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:40:46.376969 30903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bca1a9896e4556b95376caaf5070c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:46.376984 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bca1a9896e4556b95376caaf5070c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:40:46.380071 31030 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Bootstrap starting.
I20260811 02:40:46.380095 31031 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Bootstrap starting.
I20260811 02:40:46.380864 31030 tablet_bootstrap.cc:654] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:46.380909 31031 tablet_bootstrap.cc:654] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Neither blocks nor log segments found. Creating new log.
I20260811 02:40:46.381356 31030 log.cc:824] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:46.381410 31031 log.cc:824] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Log is configured to *not* fsync() on all Append() calls
I20260811 02:40:46.382196 31031 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: No bootstrap required, opened a new log
I20260811 02:40:46.382216 31030 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: No bootstrap required, opened a new log
I20260811 02:40:46.382261 31031 ts_tablet_manager.cc:1397] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:40:46.382295 31030 ts_tablet_manager.cc:1397] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:40:46.383850 31031 raft_consensus.cc:348] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.383896 31030 raft_consensus.cc:348] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.384028 31031 raft_consensus.cc:374] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:46.384033 31030 raft_consensus.cc:374] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:40:46.384052 31031 raft_consensus.cc:729] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1aa4d8345f04d859c9c66213c0cbea5, State: Initialized, Role: FOLLOWER
I20260811 02:40:46.384064 31030 raft_consensus.cc:729] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff2de8f1d074bcab5c7bea1e0cb1fea, State: Initialized, Role: FOLLOWER
I20260811 02:40:46.384187 31031 consensus_queue.cc:260] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.384188 31030 consensus_queue.cc:260] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.384510 31030 ts_tablet_manager.cc:1428] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:40:46.384630 31012 heartbeater.cc:503] Master 127.2.251.62:38957 was elected leader, sending a full tablet report...
I20260811 02:40:46.384696 31031 ts_tablet_manager.cc:1428] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:40:46.384873 30797 heartbeater.cc:503] Master 127.2.251.62:38957 was elected leader, sending a full tablet report...
W20260811 02:40:46.434916 30799 tablet.cc:2369] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:40:46.526249 31037 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:40:46.526403 31037 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.526788 31037 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691), 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623)
I20260811 02:40:46.529773 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.527181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:45525 (local address 127.2.251.2:38691)
0811 02:40:46.527254 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.527256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.527266 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:40:46.527299 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.527301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.527323 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.527389 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:46.527549 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.528033 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.528992 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.529409 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.529474 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.529487 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.529632 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.529670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 02:40:46.529773 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.527017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38691 (local address 127.2.251.3:45525)
0811 02:40:46.527110 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.527126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.527196 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.527412 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.527414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.527415 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:46.527484 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.527486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.528111 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.528112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.528892 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.528894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.529493 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.529493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.529569 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.529572 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.529617 (+ 45us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.529662 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 02:40:46.530179 30728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bca1a9896e4556b95376caaf5070c9" candidate_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" is_pre_election: true
I20260811 02:40:46.530383 30728 raft_consensus.cc:2454] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fff2de8f1d074bcab5c7bea1e0cb1fea in term 0.
I20260811 02:40:46.530599 31038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.527141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40623 (local address 127.2.251.3:49471)
0811 02:40:46.527309 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.527324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 02:40:46.527408 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:40:46.527564 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:40:46.527565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:40:46.527567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:40:46.527643 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.527645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.528563 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.528566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.529457 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:40:46.529461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.530333 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.530334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:40:46.530413 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.530418 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:40:46.530479 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 02:40:46.530520 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20260811 02:40:46.530612 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.527280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:49471 (local address 127.2.251.1:40623)
0811 02:40:46.527328 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.527330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.527404 (+ 74us) server_negotiation.cc:407] Connection header received
0811 02:40:46.527464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.527465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.527492 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.527550 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:40:46.527709 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.528355 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.529607 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.530218 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.530307 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.530385 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.530486 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.530538 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:40:46.530690 30830 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea; no voters:
I20260811 02:40:46.530833 31037 raft_consensus.cc:2781] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:40:46.530897 31037 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:40:46.530931 31037 raft_consensus.cc:3037] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:46.531029 30503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bca1a9896e4556b95376caaf5070c9" candidate_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" is_pre_election: true
I20260811 02:40:46.531147 30503 raft_consensus.cc:2454] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fff2de8f1d074bcab5c7bea1e0cb1fea in term 0.
I20260811 02:40:46.531739 31037 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.531865 31037 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [CANDIDATE]: Term 1 election: Requested vote from peers c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691), 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623)
I20260811 02:40:46.532063 30728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bca1a9896e4556b95376caaf5070c9" candidate_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1aa4d8345f04d859c9c66213c0cbea5"
I20260811 02:40:46.532147 30728 raft_consensus.cc:3037] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:46.532166 30503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bca1a9896e4556b95376caaf5070c9" candidate_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
I20260811 02:40:46.532267 30503 raft_consensus.cc:3037] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:40:46.532965 30728 raft_consensus.cc:2454] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fff2de8f1d074bcab5c7bea1e0cb1fea in term 1.
I20260811 02:40:46.533138 30503 raft_consensus.cc:2454] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fff2de8f1d074bcab5c7bea1e0cb1fea in term 1.
I20260811 02:40:46.533169 30830 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea; no voters:
I20260811 02:40:46.533258 31037 raft_consensus.cc:2781] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:40:46.533452 31037 raft_consensus.cc:686] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 LEADER]: Becoming Leader. State: Replica: fff2de8f1d074bcab5c7bea1e0cb1fea, State: Running, Role: LEADER
I20260811 02:40:46.533577 31037 consensus_queue.cc:237] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:40:46.534598 30248 catalog_manager.cc:5495] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea reported cstate change: term changed from 0 to 1, leader changed from <none> to fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3). New cstate: current_term: 1 leader_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } health_report { overall_health: UNKNOWN } } }
W20260811 02:40:46.541219 30574 tablet.cc:2369] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:40:46.564332 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.562057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.2.251.3:35273)
0811 02:40:46.562130 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 02:40:46.562132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:40:46.562139 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:40:46.562179 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:40:46.562180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:40:46.562204 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:40:46.562274 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:40:46.562389 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.562875 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.563577 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:40:46.563776 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:40:46.563832 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:40:46.563976 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:40:46.564091 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:40:46.564168 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 02:40:46.564189 (+ 21us) server_negotiation.cc:299] Negotiation successful
0811 02:40:46.564212 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:40:46.566412 30503 raft_consensus.cc:1261] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Refusing update from remote peer fff2de8f1d074bcab5c7bea1e0cb1fea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:46.566542 30728 raft_consensus.cc:1261] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Refusing update from remote peer fff2de8f1d074bcab5c7bea1e0cb1fea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:40:46.567016 31039 consensus_queue.cc:1035] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:46.567189 31037 consensus_queue.cc:1035] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:40:46.571223 31045 mvcc.cc:205] Tried to move back new op lower bound from 7317160126732890112 to 7317160126602338304. Current Snapshot: MvccSnapshot[applied={T|T < 7317160126732890112 or (T in {7317160126732890112})}]
I20260811 02:40:46.571835 31048 mvcc.cc:205] Tried to move back new op lower bound from 7317160126732890112 to 7317160126602338304. Current Snapshot: MvccSnapshot[applied={T|T < 7317160126732890112 or (T in {7317160126732890112})}]
I20260811 02:40:46.572825 30247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:46.572949 30247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:40:46.573983 30247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Sending DeleteTablet for 3 replicas of tablet 51bca1a9896e4556b95376caaf5070c9
I20260811 02:40:46.574374 30903 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bca1a9896e4556b95376caaf5070c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:46 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49244
I20260811 02:40:46.574646 30464 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bca1a9896e4556b95376caaf5070c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:46 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:44408
I20260811 02:40:46.574517 30692 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bca1a9896e4556b95376caaf5070c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:40:46 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:34264
I20260811 02:40:46.574766 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 30325
I20260811 02:40:46.574854 31063 tablet_replica.cc:330] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: stopping tablet replica
I20260811 02:40:46.574992 31063 raft_consensus.cc:2229] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:40:46.575098 31063 raft_consensus.cc:2258] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:46.583003 31063 ts_tablet_manager.cc:1905] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:46.584978 31063 ts_tablet_manager.cc:1918] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:46.585078 31063 log.cc:1196] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/51bca1a9896e4556b95376caaf5070c9
I20260811 02:40:46.585371 31063 ts_tablet_manager.cc:1939] T 51bca1a9896e4556b95376caaf5070c9 P fff2de8f1d074bcab5c7bea1e0cb1fea: Deleting consensus metadata
I20260811 02:40:46.585939 30225 catalog_manager.cc:4841] TS fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): tablet 51bca1a9896e4556b95376caaf5070c9 (table test-table [id=c6ad08d8ea684b1ebdd31505ec0bdbe8]) successfully deleted
I20260811 02:40:46.595311 31067 tablet_replica.cc:330] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: stopping tablet replica
I20260811 02:40:46.595538 31067 raft_consensus.cc:2229] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:40:46.595685 31067 raft_consensus.cc:2258] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:40:46.598124 31067 ts_tablet_manager.cc:1905] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:40:46.599838 31067 ts_tablet_manager.cc:1918] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:40:46.599938 31067 log.cc:1196] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/51bca1a9896e4556b95376caaf5070c9
I20260811 02:40:46.600277 31067 ts_tablet_manager.cc:1939] T 51bca1a9896e4556b95376caaf5070c9 P c1aa4d8345f04d859c9c66213c0cbea5: Deleting consensus metadata
I20260811 02:40:46.600795 30223 catalog_manager.cc:4841] TS c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): tablet 51bca1a9896e4556b95376caaf5070c9 (table test-table [id=c6ad08d8ea684b1ebdd31505ec0bdbe8]) successfully deleted
W20260811 02:40:46.603628 30223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:40623 (error 108)
I20260811 02:40:46.603924 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 30579
I20260811 02:40:46.616385 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:40:46.604064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:40623 (local address 127.0.0.1:44410)
0811 02:40:46.616240 (+ 12176us) negotiation.cc:107] Waiting for socket to connect
0811 02:40:46.616329 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:40623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12128}
W20260811 02:40:46.640980 30223 catalog_manager.cc:4584] TS 4e0a47ed487f4e0592201cb68d2c4a3c (127.2.251.1:40623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51bca1a9896e4556b95376caaf5070c9: Network error: Client connection negotiation failed: client connection to 127.2.251.1:40623: connect: Connection refused (error 111)
I20260811 02:40:46.651952 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 30802
I20260811 02:40:46.666198 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 30198
I20260811 02:41:06.680918 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38957
--webserver_interface=127.2.251.62
--webserver_port=46731
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:06.774991 31070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:06.775246 31070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:06.775270 31070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:06.776602 31070 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 02:41:06.776669 31070 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 02:41:06.776682 31070 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 02:41:06.776693 31070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:06.776705 31070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:06.776718 31070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:06.776729 31070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:06.776746 31070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:06.779089 31070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=46731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:06.779510 31070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:06.779776 31070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:06.786748 31082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:06.786883 31080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:06.786778 31085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:06.787417 31070 server_base.cc:1034] running on GCE node
I20260811 02:41:06.787726 31070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:06.788058 31070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:06.789247 31070 hybrid_clock.cc:648] HybridClock initialized: now 1786416066789228 us; error 39 us; skew 500 ppm
Aug 11 02:41:06 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416066, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:06.796778 31070 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:06.797258 31070 webserver.cc:460] Webserver started at http://127.2.251.62:46731/ using document root <none> and password file <none>
I20260811 02:41:06.797501 31070 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:06.797546 31070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:06.799127 31070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:06.799935 31095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:06.800158 31070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:06.800266 31070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "933ffd70c9a24a78a4bae3f8f26665a7"
format_stamp: "Formatted at 2026-08-11 02:40:24 on dist-test-slave-cvwc"
server_key: "e6c655fdc8ba14a26432930ee58a0905"
server_key_iv: "82c1997fffe1f2fed3aa6d016fc1c699"
server_key_version: "encryptionkey@0"
I20260811 02:41:06.800936 31070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:06.847328 31070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:06.850174 31070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:06.850350 31070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:06.855463 31070 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:38957
I20260811 02:41:06.855513 31166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:38957 every 8 connection(s)
I20260811 02:41:06.855973 31070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:06.857904 31167 sys_catalog.cc:263] Verifying existing consensus state
I20260811 02:41:06.859122 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap starting.
I20260811 02:41:06.860049 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 31070
I20260811 02:41:06.860329 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:40623
--local_ip_for_outbound_sockets=127.2.251.1
--tserver_master_addrs=127.2.251.62:38957
--webserver_port=41415
--webserver_interface=127.2.251.1
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:41:06.870352 31167 log.cc:824] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:06.872953 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:41:06.873303 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Bootstrap complete.
I20260811 02:41:06.875613 31167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:41:06.875854 31167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933ffd70c9a24a78a4bae3f8f26665a7, State: Initialized, Role: FOLLOWER
I20260811 02:41:06.876009 31167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:41:06.876071 31167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:06.876118 31167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:06.876166 31167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 2 FOLLOWER]: Advancing to term 3
I20260811 02:41:06.877264 31167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:41:06.877378 31167 leader_election.cc:302] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933ffd70c9a24a78a4bae3f8f26665a7; no voters:
I20260811 02:41:06.877566 31167 leader_election.cc:288] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 02:41:06.877619 31174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 02:41:06.877779 31174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [term 3 LEADER]: Becoming Leader. State: Replica: 933ffd70c9a24a78a4bae3f8f26665a7, State: Running, Role: LEADER
I20260811 02:41:06.877882 31167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:06.877876 31174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } }
I20260811 02:41:06.878204 31176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } } }
I20260811 02:41:06.878207 31179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933ffd70c9a24a78a4bae3f8f26665a7. Latest consensus state: current_term: 3 leader_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933ffd70c9a24a78a4bae3f8f26665a7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38957 } } }
I20260811 02:41:06.878321 31176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:06.878325 31179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:06.878890 31184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:06.879927 31184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:06.880057 31184 catalog_manager.cc:1259] Loaded cluster ID: 633c06f5da2c403d9e20735f8fcf8b14
I20260811 02:41:06.880095 31184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:06.882133 31184 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:06.882372 31184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Loaded TSK: 0
I20260811 02:41:06.882427 31184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 933ffd70c9a24a78a4bae3f8f26665a7: Loaded TSK: 1
F20260811 02:41:06.882668 31184 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f09457279ad google::LogMessage::Fail() at ??:0
@ 0x7f094572a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f09457274de google::LogMessage::Flush() at ??:0
@ 0x7f0945728a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0947e94104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0947e91d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f094d6811cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f094d68148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f094d681594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f094d647c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f094d651ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f09475f50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f09475ea954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f094844e6db start_thread at ??:0
@ 0x7f094422561f clone at ??:0
W20260811 02:41:06.976217 31170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:06.976451 31170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:06.976477 31170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:06.977699 31170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:06.977793 31170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:06.977828 31170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:06.980331 31170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:40623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=41415
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:06.980777 31170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:06.981092 31170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:06.988076 31241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:06.988087 31244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:06.988301 31170 server_base.cc:1034] running on GCE node
W20260811 02:41:06.988080 31246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:06.988678 31170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:06.988957 31170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:06.990188 31170 hybrid_clock.cc:648] HybridClock initialized: now 1786416066990155 us; error 39 us; skew 500 ppm
Aug 11 02:41:06 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416066, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:06.993482 31170 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:41:06.994072 31170 webserver.cc:460] Webserver started at http://127.2.251.1:41415/ using document root <none> and password file <none>
I20260811 02:41:06.994338 31170 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:06.994388 31170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:06.995774 31170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:06.996479 31255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:06.996631 31170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260811 02:41:06.996713 31170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "4e0a47ed487f4e0592201cb68d2c4a3c"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "f52a93b4ccf47a0683ff0cfecbd68f38"
server_key_iv: "b79abe09245e28afc82720a6cad5113b"
server_key_version: "encryptionkey@0"
I20260811 02:41:06.997119 31170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:07.011373 31170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:07.014314 31170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:07.014488 31170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:07.014919 31170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:07.015550 31262 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 02:41:07.017171 31170 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 02:41:07.017234 31170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260811 02:41:07.017264 31170 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 02:41:07.018064 31170 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 02:41:07.018112 31170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:07.018172 31262 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap starting.
I20260811 02:41:07.026408 31170 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:40623
I20260811 02:41:07.026599 31428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:40623 every 8 connection(s)
I20260811 02:41:07.026979 31170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:41:07.027796 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 31170
I20260811 02:41:07.028245 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:38691
--local_ip_for_outbound_sockets=127.2.251.2
--tserver_master_addrs=127.2.251.62:38957
--webserver_port=46835
--webserver_interface=127.2.251.2
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:41:07.029683 31262 log.cc:824] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:07.032322 31262 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 02:41:07.032673 31262 tablet_bootstrap.cc:492] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Bootstrap complete.
I20260811 02:41:07.033771 31262 ts_tablet_manager.cc:1397] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260811 02:41:07.034826 31262 raft_consensus.cc:348] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:07.035084 31262 raft_consensus.cc:729] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0a47ed487f4e0592201cb68d2c4a3c, State: Initialized, Role: FOLLOWER
I20260811 02:41:07.035283 31262 consensus_queue.cc:260] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:07.035579 31262 ts_tablet_manager.cc:1428] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260811 02:41:07.110064 31439 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:41:07.030303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:60279)
0811 02:41:07.032582 (+ 2279us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.032630 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:07.033305 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:07.109881 (+ 76576us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2227,"thread_start_us":105,"threads_started":1}
W20260811 02:41:07.110369 31433 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:38957 (0 consecutive failures): Network error: Failed to ping master at 127.2.251.62:38957: Client connection negotiation failed: client connection to 127.2.251.62:38957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 02:41:07.110695 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.110587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:47305)
0811 02:41:07.110635 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.110674 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260811 02:41:07.111101 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.110955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:38633)
0811 02:41:07.111035 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.111076 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 02:41:07.111222 31433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:41:07.130939 31438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:07.131182 31438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:07.131209 31438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:07.132501 31438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:07.132563 31438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:07.132592 31438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:07.134308 31438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:38691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=46835
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:07.134588 31438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:07.134833 31438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:07.141528 31450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:07.141562 31451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:07.141768 31438 server_base.cc:1034] running on GCE node
W20260811 02:41:07.141528 31455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:07.142059 31438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:07.142980 31438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:07.144243 31438 hybrid_clock.cc:648] HybridClock initialized: now 1786416067144206 us; error 77 us; skew 500 ppm
Aug 11 02:41:07 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:07.147133 31438 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:41:07.147616 31438 webserver.cc:460] Webserver started at http://127.2.251.2:46835/ using document root <none> and password file <none>
I20260811 02:41:07.147835 31438 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:07.147888 31438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:07.149333 31438 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:41:07.149999 31466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:07.150215 31438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:41:07.150315 31438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "c1aa4d8345f04d859c9c66213c0cbea5"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "cc1df7352651e0739199d6f1dece02b0"
server_key_iv: "793d386ad9af3ebb5f6e6ee4f04638a6"
server_key_version: "encryptionkey@0"
I20260811 02:41:07.150700 31438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:07.162384 31438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:07.165169 31438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:07.165328 31438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:07.165575 31438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:07.165918 31438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:07.165953 31438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:07.165983 31438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:07.166005 31438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:07.172047 31438 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:38691
I20260811 02:41:07.172094 31638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:38691 every 8 connection(s)
I20260811 02:41:07.172462 31438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:07.173565 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.173134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:35773)
0811 02:41:07.173336 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.173416 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260811 02:41:07.173954 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 31438
W20260811 02:41:07.174072 31639 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:38957 (0 consecutive failures): Network error: Failed to ping master at 127.2.251.62:38957: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
I20260811 02:41:07.174288 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:35273
--local_ip_for_outbound_sockets=127.2.251.3
--tserver_master_addrs=127.2.251.62:38957
--webserver_port=46345
--webserver_interface=127.2.251.3
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:41:07.174443 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.174267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:58051)
0811 02:41:07.174363 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.174407 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 02:41:07.174796 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.174686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:33231)
0811 02:41:07.174735 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.174771 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:41:07.174966 31639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 02:41:07.271239 31642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:07.271522 31642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:07.271562 31642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:07.273438 31642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:07.273523 31642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:07.273567 31642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:07.275736 31642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:35273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=46345
--tserver_master_addrs=127.2.251.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:07.276010 31642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:07.276223 31642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:07.283236 31650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:07.283237 31649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:07.283450 31642 server_base.cc:1034] running on GCE node
W20260811 02:41:07.283351 31653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:07.283808 31642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:07.284166 31642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:07.285362 31642 hybrid_clock.cc:648] HybridClock initialized: now 1786416067285348 us; error 37 us; skew 500 ppm
Aug 11 02:41:07 dist-test-slave-cvwc krb5kdc[29250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416067, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:07.288901 31642 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:41:07.289467 31642 webserver.cc:460] Webserver started at http://127.2.251.3:46345/ using document root <none> and password file <none>
I20260811 02:41:07.289757 31642 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:07.289825 31642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:07.291360 31642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:07.292114 31663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:07.292330 31642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:07.292425 31642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea"
format_stamp: "Formatted at 2026-08-11 02:40:25 on dist-test-slave-cvwc"
server_key: "920a3cb81d20d0f4a77061dd623ea468"
server_key_iv: "e2a070fb8287e50d55afd9b553110995"
server_key_version: "encryptionkey@0"
I20260811 02:41:07.292815 31642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:07.306965 31642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:07.309821 31642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:07.309959 31642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:07.310199 31642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:07.310527 31642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:07.310562 31642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:07.310585 31642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:07.310596 31642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:07.316464 31642 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:35273
I20260811 02:41:07.316537 31840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:35273 every 8 connection(s)
I20260811 02:41:07.316879 31642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:07.318033 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.317588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:46757)
0811 02:41:07.317789 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.317886 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
W20260811 02:41:07.318713 31841 heartbeater.cc:644] Failed to heartbeat to 127.2.251.62:38957 (0 consecutive failures): Network error: Failed to ping master at 127.2.251.62:38957: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
I20260811 02:41:07.319147 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.318992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:40115)
0811 02:41:07.319084 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.319121 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 02:41:07.319730 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:07.319513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:41313)
0811 02:41:07.319607 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:07.319684 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 02:41:07.319949 31841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 02:41:07.321375 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 31642
I20260811 02:41:08.111927 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.111645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:43843)
0811 02:41:08.111804 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:08.111853 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260811 02:41:08.171828 31846 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:08.171989 31846 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:08.172362 31846 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273), c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
I20260811 02:41:08.175532 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.175307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:40531)
0811 02:41:08.175413 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:08.175474 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
Aug 11 02:41:08 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416066, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.3@KRBTEST.COM
Aug 11 02:41:08 dist-test-slave-cvwc krb5kdc[29250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416066, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.2@KRBTEST.COM
I20260811 02:41:08.186384 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.172766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:39917 (local address 127.2.251.2:38691)
0811 02:41:08.172920 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:08.172933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:08.172959 (+ 26us) server_negotiation.cc:407] Connection header received
0811 02:41:08.173409 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:08.173412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:08.173468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:08.173614 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:08.174324 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.175261 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.176775 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.177419 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.177548 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:08.181948 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:08.181952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:08.181952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:08.181962 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:08.184067 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:08.184716 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:08.184719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:08.184719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:08.184826 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:08.185275 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:08.185277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:08.185277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:08.185647 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:08.185796 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:08.186085 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 02:41:08.186248 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260811 02:41:08.186412 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.172910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:51765 (local address 127.2.251.3:35273)
0811 02:41:08.173473 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:08.173480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:08.173496 (+ 16us) server_negotiation.cc:407] Connection header received
0811 02:41:08.173918 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:08.173920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:08.173964 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:08.174080 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:08.174393 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.175352 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.176775 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.177419 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.177553 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:08.181679 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:08.181685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:08.181686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:08.181700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:08.183974 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:08.184587 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:08.184591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:08.184592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:08.184849 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:08.185262 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:08.185264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:08.185264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:08.185602 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:08.185683 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:08.186094 (+ 411us) server_negotiation.cc:299] Negotiation successful
0811 02:41:08.186262 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":72,"threads_started":1}
I20260811 02:41:08.186434 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.172580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:38691 (local address 127.2.251.1:39917)
0811 02:41:08.172689 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:08.172701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:08.172761 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:08.173653 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:08.173656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:08.174009 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:08.174206 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:08.174212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.175356 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.175358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:08.176662 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:08.176665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.177543 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.177545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:08.177696 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:08.178454 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:08.178478 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:08.181682 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:08.184181 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:08.184182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:08.184182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:08.184496 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:08.184948 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:08.184950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:08.184951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:08.185168 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:08.185780 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:08.185782 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:08.185891 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 02:41:08.185996 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 02:41:08.186275 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":42}
I20260811 02:41:08.187178 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.172596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:35273 (local address 127.2.251.1:51765)
0811 02:41:08.172755 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:08.172764 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:08.172803 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:08.174081 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:08.174082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:08.174245 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:08.174298 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:08.174300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.175448 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.175449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:08.176662 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:08.176665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:08.177545 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:08.177546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:08.177696 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:08.178454 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:08.178478 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:08.181399 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:08.184113 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:08.184127 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:08.184131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:08.184496 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:08.185033 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:08.185033 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:08.185034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:08.185163 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:08.185691 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:08.185699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:08.185886 (+ 187us) client_negotiation.cc:769] Sending connection context
0811 02:41:08.185997 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 02:41:08.186365 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":234,"thread_start_us":81,"threads_started":1}
W20260811 02:41:08.188151 31258 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
W20260811 02:41:08.188303 31256 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
I20260811 02:41:08.188359 31256 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0a47ed487f4e0592201cb68d2c4a3c; no voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea
I20260811 02:41:08.188489 31846 raft_consensus.cc:2726] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:41:08.320662 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:08.320379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:60343)
0811 02:41:08.320560 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:08.320613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260811 02:41:09.112864 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:09.112583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:53719)
0811 02:41:09.112757 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:09.112810 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260811 02:41:09.176278 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:09.176017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:34199)
0811 02:41:09.176171 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:09.176223 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260811 02:41:09.321576 31858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:09.321214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:54109)
0811 02:41:09.321439 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:09.321491 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260811 02:41:10.113835 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:10.113564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:45923)
0811 02:41:10.113704 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:10.113783 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260811 02:41:10.177070 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:10.176838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:54365)
0811 02:41:10.176977 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:10.177020 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260811 02:41:10.224591 31861 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:10.224692 31861 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:10.224864 31861 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273), c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
W20260811 02:41:10.225278 31258 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
W20260811 02:41:10.225337 31256 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
I20260811 02:41:10.225454 31256 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0a47ed487f4e0592201cb68d2c4a3c; no voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea
I20260811 02:41:10.225586 31861 raft_consensus.cc:2726] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:41:10.322646 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:10.322215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:59905)
0811 02:41:10.322555 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:10.322596 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":76,"threads_started":1}
I20260811 02:41:11.114674 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:11.114422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:60181)
0811 02:41:11.114578 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:11.114623 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260811 02:41:11.177948 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:11.177566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:37599)
0811 02:41:11.177815 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:11.177875 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
I20260811 02:41:11.323395 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:11.323131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:38649)
0811 02:41:11.323297 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:11.323344 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260811 02:41:11.851303 31867 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:11.851420 31867 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:11.851589 31867 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273), c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
W20260811 02:41:11.851989 31258 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
W20260811 02:41:11.852082 31256 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
I20260811 02:41:11.852128 31256 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0a47ed487f4e0592201cb68d2c4a3c; no voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea
I20260811 02:41:11.852187 31867 raft_consensus.cc:2726] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:41:12.115566 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:12.115271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:38417)
0811 02:41:12.115461 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:12.115513 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260811 02:41:12.179127 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:12.178596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:54107)
0811 02:41:12.178842 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:12.179047 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260811 02:41:12.324404 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:12.324073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:56027)
0811 02:41:12.324291 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:12.324334 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":138,"threads_started":1}
I20260811 02:41:13.116456 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:13.116188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:54817)
0811 02:41:13.116346 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:13.116401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260811 02:41:13.180155 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:13.179844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:35565)
0811 02:41:13.180025 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:13.180082 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260811 02:41:13.325654 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:13.325285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:39287)
0811 02:41:13.325544 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:13.325596 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260811 02:41:14.117377 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:14.117103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:58603)
0811 02:41:14.117260 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:14.117327 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260811 02:41:14.181190 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:14.180906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:47567)
0811 02:41:14.181080 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:14.181127 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260811 02:41:14.326591 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:14.326252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:44125)
0811 02:41:14.326410 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:14.326450 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260811 02:41:15.118292 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:15.118016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:42981)
0811 02:41:15.118178 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:15.118231 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260811 02:41:15.182018 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:15.181781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:41395)
0811 02:41:15.181920 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:15.181957 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260811 02:41:15.327754 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:15.327440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:49093)
0811 02:41:15.327640 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:15.327697 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260811 02:41:16.119405 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:16.119045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:37755)
0811 02:41:16.119280 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:16.119344 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260811 02:41:16.182865 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:16.182535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:45531)
0811 02:41:16.182701 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:16.182748 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260811 02:41:16.207285 31887 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:16.207660 31887 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:16.207944 31887 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273), c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
W20260811 02:41:16.208415 31256 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
W20260811 02:41:16.208546 31258 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
I20260811 02:41:16.208622 31258 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0a47ed487f4e0592201cb68d2c4a3c; no voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea
I20260811 02:41:16.208730 31887 raft_consensus.cc:2726] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:41:16.328873 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:16.328412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:54619)
0811 02:41:16.328733 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:16.328810 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":224,"threads_started":1}
I20260811 02:41:17.120566 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:17.120121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:51991)
0811 02:41:17.120396 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:17.120483 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260811 02:41:17.184113 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:17.183836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:58955)
0811 02:41:17.184007 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:17.184056 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260811 02:41:17.329648 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:17.329425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:37301)
0811 02:41:17.329559 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:17.329593 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260811 02:41:18.121634 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:18.121324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:53749)
0811 02:41:18.121526 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:18.121579 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260811 02:41:18.185509 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:18.184934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:51725)
0811 02:41:18.185229 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:18.185453 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":76,"threads_started":1}
I20260811 02:41:18.330550 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:18.330199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:54433)
0811 02:41:18.330420 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:18.330476 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260811 02:41:19.122681 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:19.122252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:51001)
0811 02:41:19.122498 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:19.122609 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260811 02:41:19.186436 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:19.186183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:58333)
0811 02:41:19.186328 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:19.186380 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260811 02:41:19.331619 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:19.331271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:41581)
0811 02:41:19.331501 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:19.331564 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260811 02:41:20.123660 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:20.123393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:40689)
0811 02:41:20.123558 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:20.123608 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260811 02:41:20.187278 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:20.187019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:47851)
0811 02:41:20.187173 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:20.187224 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260811 02:41:20.332527 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:20.332233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:39007)
0811 02:41:20.332424 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:20.332472 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260811 02:41:20.709183 31917 raft_consensus.cc:482] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:20.709307 31917 raft_consensus.cc:504] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff2de8f1d074bcab5c7bea1e0cb1fea" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 35273 } } peers { permanent_uuid: "c1aa4d8345f04d859c9c66213c0cbea5" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 38691 } } peers { permanent_uuid: "4e0a47ed487f4e0592201cb68d2c4a3c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 40623 } }
I20260811 02:41:20.709504 31917 leader_election.cc:288] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273), c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691)
W20260811 02:41:20.709946 31256 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c1aa4d8345f04d859c9c66213c0cbea5 (127.2.251.2:38691): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
W20260811 02:41:20.710083 31258 leader_election.cc:341] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fff2de8f1d074bcab5c7bea1e0cb1fea (127.2.251.3:35273): Not found: Tablet not found: 51bca1a9896e4556b95376caaf5070c9
I20260811 02:41:20.710152 31258 leader_election.cc:302] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0a47ed487f4e0592201cb68d2c4a3c; no voters: c1aa4d8345f04d859c9c66213c0cbea5, fff2de8f1d074bcab5c7bea1e0cb1fea
I20260811 02:41:20.710217 31917 raft_consensus.cc:2726] T 51bca1a9896e4556b95376caaf5070c9 P 4e0a47ed487f4e0592201cb68d2c4a3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 02:41:21.124730 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:21.124284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:32867)
0811 02:41:21.124632 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:21.124678 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"spinlock_wait_cycles":179072,"thread_start_us":155,"threads_started":1}
I20260811 02:41:21.188202 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:21.187974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:44141)
0811 02:41:21.188102 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:21.188148 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260811 02:41:21.333334 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:21.333071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:48845)
0811 02:41:21.333241 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:21.333278 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260811 02:41:22.125576 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:22.125308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.1:43231)
0811 02:41:22.125483 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:22.125526 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260811 02:41:22.189086 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:22.188844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.2:44011)
0811 02:41:22.188990 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:22.189034 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260811 02:41:22.323428 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 31170
I20260811 02:41:22.340687 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:22.339650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38957 (local address 127.2.251.3:43085)
0811 02:41:22.340573 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:22.340626 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":816,"threads_started":1}
I20260811 02:41:22.344858 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 31438
I20260811 02:41:22.373428 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 31642
2026-08-11T02:41:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59045 ms)
[----------] 30 tests from SecurityITest (251067 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:22 dist-test-slave-cvwc krb5kdc[31935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:22 dist-test-slave-cvwc krb5kdc[31935](info): set up 2 sockets
Aug 11 02:41:22 dist-test-slave-cvwc krb5kdc[31935](info): commencing operation
krb5kdc: starting...
W20260811 02:41:23.590202 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:23 dist-test-slave-cvwc krb5kdc[31935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:23.728629 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:35145
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:23.822589 31955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:23.822947 31955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:23.822990 31955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:23.825205 31955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:23.825310 31955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:23.825335 31955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:23.825356 31955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:23.825372 31955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:23.827394 31955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:23.827718 31955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:23.827976 31955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:23.833901 31966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:23.833915 31963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:23.833959 31955 server_base.cc:1034] running on GCE node
W20260811 02:41:23.833925 31970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:23.834383 31955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:23.834712 31955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:23.835893 31955 hybrid_clock.cc:648] HybridClock initialized: now 1786416083835875 us; error 48 us; skew 500 ppm
Aug 11 02:41:23 dist-test-slave-cvwc krb5kdc[31935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416083, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:23.843547 31955 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:23.844094 31955 webserver.cc:460] Webserver started at http://127.2.251.62:45183/ using document root <none> and password file <none>
I20260811 02:41:23.844327 31955 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:23.844406 31955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:23.844538 31955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:23.845827 31955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "0c63b542ac304632be1ceb030bc83a14"
format_stamp: "Formatted at 2026-08-11 02:41:23 on dist-test-slave-cvwc"
server_key: "cbb5ea563196eeae9eaf111bee30c736"
server_key_iv: "fd63897deaa5ec7952dd6f3ef7970972"
server_key_version: "encryptionkey@0"
I20260811 02:41:23.846165 31955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "0c63b542ac304632be1ceb030bc83a14"
format_stamp: "Formatted at 2026-08-11 02:41:23 on dist-test-slave-cvwc"
server_key: "cbb5ea563196eeae9eaf111bee30c736"
server_key_iv: "fd63897deaa5ec7952dd6f3ef7970972"
server_key_version: "encryptionkey@0"
I20260811 02:41:23.847513 31955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:23.848290 31986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:23.848482 31955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:41:23.848572 31955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "0c63b542ac304632be1ceb030bc83a14"
format_stamp: "Formatted at 2026-08-11 02:41:23 on dist-test-slave-cvwc"
server_key: "cbb5ea563196eeae9eaf111bee30c736"
server_key_iv: "fd63897deaa5ec7952dd6f3ef7970972"
server_key_version: "encryptionkey@0"
I20260811 02:41:23.848650 31955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:23.872138 31955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:23.874452 31955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:23.874579 31955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:23.878966 31955 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:35145
I20260811 02:41:23.879017 32077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:35145 every 8 connection(s)
I20260811 02:41:23.879384 31955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:23.880313 32082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:23.883018 32082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14: Bootstrap starting.
I20260811 02:41:23.883674 32082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:23.884084 32082 log.cc:824] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:23.885249 32082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14: No bootstrap required, opened a new log
I20260811 02:41:23.886473 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 31955
I20260811 02:41:23.886792 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:23.887149 3052 external_mini_cluster.cc:1428] Setting key e19fc07c1bbcc484b4853b31c41aed1c
I20260811 02:41:23.887863 32082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c63b542ac304632be1ceb030bc83a14" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35145 } }
I20260811 02:41:23.888214 32082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:23.888304 32082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c63b542ac304632be1ceb030bc83a14, State: Initialized, Role: FOLLOWER
I20260811 02:41:23.888458 32082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c63b542ac304632be1ceb030bc83a14" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35145 } }
I20260811 02:41:23.888533 32082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:23.888558 32082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:23.888609 32082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:23.889654 32082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c63b542ac304632be1ceb030bc83a14" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35145 } }
I20260811 02:41:23.889793 32082 leader_election.cc:302] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c63b542ac304632be1ceb030bc83a14; no voters:
I20260811 02:41:23.889966 32082 leader_election.cc:288] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:23.890002 32091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:23.890165 32091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [term 1 LEADER]: Becoming Leader. State: Replica: 0c63b542ac304632be1ceb030bc83a14, State: Running, Role: LEADER
I20260811 02:41:23.890273 32082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:23.890309 32091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c63b542ac304632be1ceb030bc83a14" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35145 } }
I20260811 02:41:23.890704 32092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c63b542ac304632be1ceb030bc83a14. Latest consensus state: current_term: 1 leader_uuid: "0c63b542ac304632be1ceb030bc83a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c63b542ac304632be1ceb030bc83a14" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35145 } } }
I20260811 02:41:23.890715 32095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c63b542ac304632be1ceb030bc83a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c63b542ac304632be1ceb030bc83a14" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 35145 } } }
I20260811 02:41:23.890923 32095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:23.891104 32092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:23.891431 32107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:23.892027 32107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:23.893934 32107 catalog_manager.cc:1347] Generated new cluster ID: 55c28ae14bdf4ce9944a5e2000ab244b
I20260811 02:41:23.893992 32107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:41:23 dist-test-slave-cvwc krb5kdc[31935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:23.900183 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:23.887920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.2.251.62:35145)
0811 02:41:23.888234 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:23.888246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:23.888286 (+ 40us) server_negotiation.cc:407] Connection header received
0811 02:41:23.888422 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:23.888431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:23.888496 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:23.888641 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:23.889246 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:23.890468 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:23.891524 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:23.892175 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:23.892298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:23.895303 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:23.895308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:23.895308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:23.895323 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:23.897388 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:23.897799 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:23.897803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:23.897803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:23.898641 (+ 838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:23.898872 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:23.898873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:23.898874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:23.899237 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:23.899424 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:23.899761 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 02:41:23.899936 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":196,"threads_started":1}
I20260811 02:41:23.908957 32107 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:23.909822 32107 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:23.913198 32107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14: Generated new TSK 0
I20260811 02:41:23.913475 32107 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:41:23.952658 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35145
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:24.045928 32122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:24.046172 32122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:24.046211 32122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:24.047514 32122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:24.047571 32122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:24.047613 32122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:24.049304 32122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:24.049578 32122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:24.049860 32122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:24.055692 32137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:24.055703 32145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:24.055703 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.055873 32122 server_base.cc:1034] running on GCE node
I20260811 02:41:24.056196 32122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:24.056470 32122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:24.057641 32122 hybrid_clock.cc:648] HybridClock initialized: now 1786416084057623 us; error 30 us; skew 500 ppm
Aug 11 02:41:24 dist-test-slave-cvwc krb5kdc[31935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416084, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:24.060510 32122 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:41:24.061030 32122 webserver.cc:460] Webserver started at http://127.2.251.1:40459/ using document root <none> and password file <none>
I20260811 02:41:24.061251 32122 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:24.061301 32122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:24.061444 32122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:24.062628 32122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "6f720b03a8b94c91be158ad0151d9259"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "36fd031887313bb8e980c61f48e8f8a8"
server_key_iv: "f73fca9170a80245185803538530e87a"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.062986 32122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "6f720b03a8b94c91be158ad0151d9259"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "36fd031887313bb8e980c61f48e8f8a8"
server_key_iv: "f73fca9170a80245185803538530e87a"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.064427 32122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:41:24.065268 32162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.065478 32122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:41:24.065572 32122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "6f720b03a8b94c91be158ad0151d9259"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "36fd031887313bb8e980c61f48e8f8a8"
server_key_iv: "f73fca9170a80245185803538530e87a"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.065644 32122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:24.089260 32122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:24.091828 32122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:24.091995 32122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:24.092264 32122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:24.092654 32122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:24.092700 32122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.092737 32122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:24.092761 32122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.098728 32122 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:45263
I20260811 02:41:24.098778 32341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:45263 every 8 connection(s)
I20260811 02:41:24.099134 32122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:41:24.099846 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 32122
I20260811 02:41:24.099952 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:41:24.100184 3052 external_mini_cluster.cc:1428] Setting key 1cd72932ad1b1192c3aaec3562c2d282
Aug 11 02:41:24 dist-test-slave-cvwc krb5kdc[31935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416084, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:24.110126 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.100047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35145 (local address 127.2.251.1:60749)
0811 02:41:24.100227 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.100278 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.100786 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.101038 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.101041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.101641 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:24.101890 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.101896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.102567 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.102568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.103385 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.103388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.103881 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.103882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.104008 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.104557 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:24.104566 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:24.106201 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:24.108197 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:24.108208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:24.108211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:24.108501 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:24.108811 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:24.108812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:24.108813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:24.108954 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:24.109443 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:24.109451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:24.109603 (+ 152us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.109708 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.109927 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260811 02:41:24.110219 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.100099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:60749 (local address 127.2.251.62:35145)
0811 02:41:24.100152 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.100154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.100367 (+ 213us) server_negotiation.cc:407] Connection header received
0811 02:41:24.100884 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.100885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.100900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.100949 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:24.101979 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.102479 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.103470 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.103761 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.103828 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.106342 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:24.106344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:24.106344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:24.106358 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:24.108070 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.108638 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.108640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.108640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.108713 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.109114 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.109115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.109115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.109284 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:24.109375 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.109903 (+ 528us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.110073 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:41:24.110948 32342 heartbeater.cc:347] Connected to a master server at 127.2.251.62:35145
I20260811 02:41:24.111129 32342 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:24.111512 32342 heartbeater.cc:511] Master 127.2.251.62:35145 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
I20260811 02:41:24.112303 32022 ts_manager.cc:194] Registered new tserver with Master: 6f720b03a8b94c91be158ad0151d9259 (127.2.251.1:45263)
I20260811 02:41:24.112965 32022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:60749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:41:24.134172 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35145
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:24.222571 32350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:24.222811 32350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:24.222836 32350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:24.224073 32350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:24.224123 32350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:24.224150 32350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:24.225692 32350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:24.225976 32350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:24.226220 32350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:24.231827 32356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.231987 32350 server_base.cc:1034] running on GCE node
W20260811 02:41:24.231833 32360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:24.231848 32357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.232362 32350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:24.232596 32350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:24.233775 32350 hybrid_clock.cc:648] HybridClock initialized: now 1786416084233752 us; error 34 us; skew 500 ppm
Aug 11 02:41:24 dist-test-slave-cvwc krb5kdc[31935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416084, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:24.236996 32350 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:41:24.237506 32350 webserver.cc:460] Webserver started at http://127.2.251.2:45007/ using document root <none> and password file <none>
I20260811 02:41:24.237783 32350 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:24.237843 32350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:24.237990 32350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:24.239138 32350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "9f69c9cafdf04744808ccc98e91e9282"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "5f860d27248bf2cf838b8831109d1f42"
server_key_iv: "eed3aa5a0aca0b6405a71669157542d8"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.239535 32350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "9f69c9cafdf04744808ccc98e91e9282"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "5f860d27248bf2cf838b8831109d1f42"
server_key_iv: "eed3aa5a0aca0b6405a71669157542d8"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.241063 32350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:24.242049 32373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.242322 32350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:24.242429 32350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "9f69c9cafdf04744808ccc98e91e9282"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "5f860d27248bf2cf838b8831109d1f42"
server_key_iv: "eed3aa5a0aca0b6405a71669157542d8"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.242503 32350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:24.278738 32350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:24.280963 32350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:24.281128 32350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:24.281387 32350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:24.281731 32350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:24.281769 32350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.281800 32350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:24.281834 32350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.288239 32350 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:37391
I20260811 02:41:24.288293 32568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:37391 every 8 connection(s)
I20260811 02:41:24.288623 32350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:24.290894 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 32350
I20260811 02:41:24.290989 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:41:24.291182 3052 external_mini_cluster.cc:1428] Setting key 75ac270d0ea1d8e5a9a1a21b3ab73568
Aug 11 02:41:24 dist-test-slave-cvwc krb5kdc[31935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416084, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:24.299425 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.289413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:58801 (local address 127.2.251.62:35145)
0811 02:41:24.289472 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.289474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.289624 (+ 150us) server_negotiation.cc:407] Connection header received
0811 02:41:24.290166 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.290173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.290182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.290261 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:24.291661 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.292320 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.293378 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.293657 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.293759 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.296275 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:24.296277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:24.296277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:24.296284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:24.297622 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.298189 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.298190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.298191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.298249 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.298573 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.298574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.298574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.298714 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:24.298780 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.299236 (+ 456us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.299329 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 02:41:24.299731 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.289346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35145 (local address 127.2.251.2:58801)
0811 02:41:24.289504 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.289547 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.290079 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.290487 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.290494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.291300 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:24.291572 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.291580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.292428 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.292429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.293282 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.293285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.293770 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.293771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.293947 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.294500 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:24.294510 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:24.296142 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:24.297733 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:24.297750 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:24.297753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:24.298085 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:24.298314 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:24.298315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:24.298315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:24.298472 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:24.298810 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:24.298819 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:24.299046 (+ 227us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.299182 (+ 136us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.299472 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260811 02:41:24.300750 32569 heartbeater.cc:347] Connected to a master server at 127.2.251.62:35145
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
I20260811 02:41:24.300917 32569 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:24.301195 32569 heartbeater.cc:511] Master 127.2.251.62:35145 requested a full tablet report, sending...
I20260811 02:41:24.301826 32022 ts_manager.cc:194] Registered new tserver with Master: 9f69c9cafdf04744808ccc98e91e9282 (127.2.251.2:37391)
I20260811 02:41:24.302232 32022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:58801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:41:24.322849 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35145
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:24.415463 32575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:24.415700 32575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:24.415727 32575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:24.417002 32575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:24.417058 32575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:24.417089 32575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:24.418744 32575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:35145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:24.419039 32575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:24.419274 32575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:24.424700 32582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:24.424753 32581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.424935 32575 server_base.cc:1034] running on GCE node
W20260811 02:41:24.424934 32584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.425189 32575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:24.425462 32575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:24.426617 32575 hybrid_clock.cc:648] HybridClock initialized: now 1786416084426607 us; error 37 us; skew 500 ppm
Aug 11 02:41:24 dist-test-slave-cvwc krb5kdc[31935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416084, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:24.429435 32575 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:41:24.430006 32575 webserver.cc:460] Webserver started at http://127.2.251.3:44379/ using document root <none> and password file <none>
I20260811 02:41:24.430245 32575 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:24.430310 32575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:24.430456 32575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:24.431609 32575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "be7d1f019e184ddcace6a21f9ec7391d"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "ec58b1ccc5c184ad1faee172657e9dc1"
server_key_iv: "4a1358e07e348b1b879e0ddd046246bc"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.431984 32575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "be7d1f019e184ddcace6a21f9ec7391d"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "ec58b1ccc5c184ad1faee172657e9dc1"
server_key_iv: "4a1358e07e348b1b879e0ddd046246bc"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.433853 32575 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 02:41:24.434746 32598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.435164 32575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:24.435269 32575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "be7d1f019e184ddcace6a21f9ec7391d"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "ec58b1ccc5c184ad1faee172657e9dc1"
server_key_iv: "4a1358e07e348b1b879e0ddd046246bc"
server_key_version: "encryptionkey@0"
I20260811 02:41:24.435359 32575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:24.446993 32575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:24.449344 32575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:24.449509 32575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:24.449839 32575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:24.450170 32575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:24.450206 32575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.450237 32575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:24.450260 32575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:24.456439 32575 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:46143
I20260811 02:41:24.456523 32767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:46143 every 8 connection(s)
I20260811 02:41:24.456848 32575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:24.459587 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 32575
I20260811 02:41:24.459690 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:41:24.459872 3052 external_mini_cluster.cc:1428] Setting key c6729be6efebae873584cb584f54b7eb
Aug 11 02:41:24 dist-test-slave-cvwc krb5kdc[31935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416084, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:24.466420 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.457702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:36517 (local address 127.2.251.62:35145)
0811 02:41:24.457793 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.457796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.458000 (+ 204us) server_negotiation.cc:407] Connection header received
0811 02:41:24.458556 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.458558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.458569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.458638 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:24.459733 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.460310 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.461417 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.461617 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.461674 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.463873 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:24.463875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:24.463875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:24.463883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:24.465076 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.465460 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.465461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.465461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.465517 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.465765 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.465766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.465766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.465892 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:24.465947 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.466228 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.466312 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 02:41:24.466656 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.457657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:35145 (local address 127.2.251.3:36517)
0811 02:41:24.457839 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.457879 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.458411 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.458760 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.458764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.459396 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:24.459612 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.459617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.460400 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.460401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.461317 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.461321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.461689 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.461690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.461837 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.462364 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:24.462373 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:24.463767 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:24.465151 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:24.465158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:24.465161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:24.465391 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:24.465576 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:24.465576 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:24.465577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:24.465682 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:24.465948 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:24.465956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:24.466096 (+ 140us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.466180 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.466355 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260811 02:41:24.467172 300 heartbeater.cc:347] Connected to a master server at 127.2.251.62:35145
I20260811 02:41:24.467275 300 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:24.467466 300 heartbeater.cc:511] Master 127.2.251.62:35145 requested a full tablet report, sending...
I20260811 02:41:24.467840 32022 ts_manager.cc:194] Registered new tserver with Master: be7d1f019e184ddcace6a21f9ec7391d (127.2.251.3:46143)
I20260811 02:41:24.468381 32022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:36517
I20260811 02:41:24.471866 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:41:24.479246 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.473307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.2.251.62:35145)
0811 02:41:24.473393 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.473395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.473454 (+ 59us) server_negotiation.cc:407] Connection header received
0811 02:41:24.473532 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.473534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.473544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.473585 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:24.474192 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.474692 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.475404 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.475590 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.475644 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.476819 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:24.476820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:24.476821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:24.476828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:24.478121 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.478445 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.478446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.478447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.478510 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:24.478673 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:24.478674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:24.478674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:24.478833 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:24.478933 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.479053 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.479148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 02:41:24.481801 32022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:41:24.484174 32022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:41:24.491869 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.487070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.2.251.2:37391)
0811 02:41:24.487528 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.487531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.487539 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:24.487574 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.487576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.487589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.487657 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:24.488024 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.489163 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.490785 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.491427 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.491508 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.491578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.491726 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.491772 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":95,"threads_started":1}
I20260811 02:41:24.491885 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.486935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:37391 (local address 127.0.0.1:32864)
0811 02:41:24.487172 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.487214 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.487289 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.487724 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.487728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.487732 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:24.487919 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.487924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.489293 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.489294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.490710 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.490712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.491521 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.491522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.491614 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.491630 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.491698 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.491766 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
I20260811 02:41:24.492148 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.486960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45263 (local address 127.0.0.1:43704)
0811 02:41:24.487863 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.487872 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.487916 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.488223 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.488224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.488227 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:24.488280 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.488282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.489466 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.489467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.490562 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.490565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.491652 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.491653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.491726 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.491730 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.491778 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.491825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":6,"thread_start_us":112,"threads_started":1}
I20260811 02:41:24.492081 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.487265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.2.251.1:45263)
0811 02:41:24.487455 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.487459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.488067 (+ 608us) server_negotiation.cc:407] Connection header received
0811 02:41:24.488118 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.488120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.488136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.488213 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:24.488373 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.489378 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.490643 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.491480 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.491554 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.491611 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.491931 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.491984 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260811 02:41:24.492179 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.487270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.2.251.3:46143)
0811 02:41:24.487455 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.487460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.487897 (+ 437us) server_negotiation.cc:407] Connection header received
0811 02:41:24.488010 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.488021 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.488051 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.488229 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:24.488459 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.489149 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.490728 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.491479 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.491553 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.491611 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.491910 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.492102 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260811 02:41:24.492673 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.486960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:46143 (local address 127.0.0.1:59654)
0811 02:41:24.487741 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.487750 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.487796 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.488325 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.488326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.488328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:24.488391 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.488393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.489249 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.489250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.490629 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.490633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.491650 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.491652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.491733 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.491736 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.491777 (+ 41us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.491824 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":108,"thread_start_us":118,"threads_started":1}
I20260811 02:41:24.493057 32230 tablet_service.cc:1467] Processing CreateTablet for tablet e9eb6721d4854f2684b6ae21d287d981 (DEFAULT_TABLE table=test-table [id=87f61be9a44447788ae9891550e5dfe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:24.493057 32454 tablet_service.cc:1467] Processing CreateTablet for tablet e9eb6721d4854f2684b6ae21d287d981 (DEFAULT_TABLE table=test-table [id=87f61be9a44447788ae9891550e5dfe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:24.493494 32230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9eb6721d4854f2684b6ae21d287d981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:24.493494 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9eb6721d4854f2684b6ae21d287d981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:24.497246 328 tablet_bootstrap.cc:492] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: Bootstrap starting.
I20260811 02:41:24.497541 327 tablet_bootstrap.cc:492] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Bootstrap starting.
I20260811 02:41:24.497898 32678 tablet_service.cc:1467] Processing CreateTablet for tablet e9eb6721d4854f2684b6ae21d287d981 (DEFAULT_TABLE table=test-table [id=87f61be9a44447788ae9891550e5dfe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:24.498236 32678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9eb6721d4854f2684b6ae21d287d981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:24.498301 327 tablet_bootstrap.cc:654] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:24.498929 327 log.cc:824] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:24.499827 327 tablet_bootstrap.cc:492] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: No bootstrap required, opened a new log
I20260811 02:41:24.499869 328 tablet_bootstrap.cc:654] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:24.500029 327 ts_tablet_manager.cc:1397] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:41:24.500510 328 log.cc:824] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:24.501255 328 tablet_bootstrap.cc:492] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: No bootstrap required, opened a new log
I20260811 02:41:24.501339 328 ts_tablet_manager.cc:1397] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 02:41:24.502061 332 tablet_bootstrap.cc:492] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Bootstrap starting.
I20260811 02:41:24.502642 327 raft_consensus.cc:348] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.502878 327 raft_consensus.cc:374] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:24.502903 327 raft_consensus.cc:729] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f69c9cafdf04744808ccc98e91e9282, State: Initialized, Role: FOLLOWER
I20260811 02:41:24.503006 332 tablet_bootstrap.cc:654] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:24.503180 327 consensus_queue.cc:260] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.503397 328 raft_consensus.cc:348] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.503620 328 raft_consensus.cc:374] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:24.503659 328 raft_consensus.cc:729] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f720b03a8b94c91be158ad0151d9259, State: Initialized, Role: FOLLOWER
I20260811 02:41:24.503661 327 ts_tablet_manager.cc:1428] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 02:41:24.503674 32569 heartbeater.cc:503] Master 127.2.251.62:35145 was elected leader, sending a full tablet report...
I20260811 02:41:24.503801 328 consensus_queue.cc:260] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.504185 32342 heartbeater.cc:503] Master 127.2.251.62:35145 was elected leader, sending a full tablet report...
I20260811 02:41:24.504444 328 ts_tablet_manager.cc:1428] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:41:24.504442 332 log.cc:824] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:24.505306 332 tablet_bootstrap.cc:492] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: No bootstrap required, opened a new log
I20260811 02:41:24.505380 332 ts_tablet_manager.cc:1397] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:41:24.506867 332 raft_consensus.cc:348] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.507028 332 raft_consensus.cc:374] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:24.507064 332 raft_consensus.cc:729] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7d1f019e184ddcace6a21f9ec7391d, State: Initialized, Role: FOLLOWER
I20260811 02:41:24.507192 332 consensus_queue.cc:260] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.507475 332 ts_tablet_manager.cc:1428] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:41:24.507586 300 heartbeater.cc:503] Master 127.2.251.62:35145 was elected leader, sending a full tablet report...
W20260811 02:41:24.539259 32570 tablet.cc:2369] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:41:24.599988 32343 tablet.cc:2369] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:41:24.707669 301 tablet.cc:2369] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:24.716377 336 raft_consensus.cc:482] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:24.716557 336 raft_consensus.cc:504] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.716943 336 leader_election.cc:288] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f69c9cafdf04744808ccc98e91e9282 (127.2.251.2:37391), be7d1f019e184ddcace6a21f9ec7391d (127.2.251.3:46143)
I20260811 02:41:24.721248 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.717207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:41113 (local address 127.2.251.3:46143)
0811 02:41:24.717259 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.717261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.717314 (+ 53us) server_negotiation.cc:407] Connection header received
0811 02:41:24.717402 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.717403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.717415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.717464 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:24.717730 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.718509 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.720146 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.720757 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.720840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.720853 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.721118 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.721168 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:41:24.721248 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.717207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:42567 (local address 127.2.251.2:37391)
0811 02:41:24.717260 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.717262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.717327 (+ 65us) server_negotiation.cc:407] Connection header received
0811 02:41:24.717392 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.717393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.717406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.717464 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:24.717733 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.718515 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.720147 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.720760 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.720840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.720853 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.721118 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.721168 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:41:24.721262 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.717125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:46143 (local address 127.2.251.1:41113)
0811 02:41:24.717251 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.717259 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.717328 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.717475 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.717476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.717479 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:24.717604 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.717605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.718639 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.718640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.720024 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.720029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.720896 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.720897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.720998 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.721006 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.721079 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.721140 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":51,"thread_start_us":65,"threads_started":1}
I20260811 02:41:24.721280 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.717111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:37391 (local address 127.2.251.1:42567)
0811 02:41:24.717245 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.717258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:24.717328 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:24.717474 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:24.717476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:24.717477 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:24.717592 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.717595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.718639 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.718640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.720025 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:24.720029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.720894 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.720896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:24.720998 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.721007 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 02:41:24.721079 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 02:41:24.721140 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":118,"thread_start_us":67,"threads_started":1}
I20260811 02:41:24.721689 32493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eb6721d4854f2684b6ae21d287d981" candidate_uuid: "6f720b03a8b94c91be158ad0151d9259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f69c9cafdf04744808ccc98e91e9282" is_pre_election: true
I20260811 02:41:24.721742 32704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eb6721d4854f2684b6ae21d287d981" candidate_uuid: "6f720b03a8b94c91be158ad0151d9259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7d1f019e184ddcace6a21f9ec7391d" is_pre_election: true
I20260811 02:41:24.721905 32704 raft_consensus.cc:2454] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f720b03a8b94c91be158ad0151d9259 in term 0.
I20260811 02:41:24.721904 32493 raft_consensus.cc:2454] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f720b03a8b94c91be158ad0151d9259 in term 0.
I20260811 02:41:24.722189 32166 leader_election.cc:302] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f720b03a8b94c91be158ad0151d9259, 9f69c9cafdf04744808ccc98e91e9282; no voters:
I20260811 02:41:24.722358 336 raft_consensus.cc:2781] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:24.722438 336 raft_consensus.cc:482] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:24.722465 336 raft_consensus.cc:3037] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:24.723237 336 raft_consensus.cc:504] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.723407 336 leader_election.cc:288] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [CANDIDATE]: Term 1 election: Requested vote from peers 9f69c9cafdf04744808ccc98e91e9282 (127.2.251.2:37391), be7d1f019e184ddcace6a21f9ec7391d (127.2.251.3:46143)
I20260811 02:41:24.723613 32493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eb6721d4854f2684b6ae21d287d981" candidate_uuid: "6f720b03a8b94c91be158ad0151d9259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f69c9cafdf04744808ccc98e91e9282"
I20260811 02:41:24.723663 32704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eb6721d4854f2684b6ae21d287d981" candidate_uuid: "6f720b03a8b94c91be158ad0151d9259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7d1f019e184ddcace6a21f9ec7391d"
I20260811 02:41:24.723697 32493 raft_consensus.cc:3037] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:24.723759 32704 raft_consensus.cc:3037] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:24.724500 32493 raft_consensus.cc:2454] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f720b03a8b94c91be158ad0151d9259 in term 1.
I20260811 02:41:24.724692 32166 leader_election.cc:302] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f720b03a8b94c91be158ad0151d9259, 9f69c9cafdf04744808ccc98e91e9282; no voters:
I20260811 02:41:24.724685 32704 raft_consensus.cc:2454] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f720b03a8b94c91be158ad0151d9259 in term 1.
I20260811 02:41:24.724795 336 raft_consensus.cc:2781] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:24.724949 336 raft_consensus.cc:686] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [term 1 LEADER]: Becoming Leader. State: Replica: 6f720b03a8b94c91be158ad0151d9259, State: Running, Role: LEADER
I20260811 02:41:24.725082 336 consensus_queue.cc:237] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } }
I20260811 02:41:24.726048 32021 catalog_manager.cc:5495] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f720b03a8b94c91be158ad0151d9259 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "6f720b03a8b94c91be158ad0151d9259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f720b03a8b94c91be158ad0151d9259" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 45263 } health_report { overall_health: HEALTHY } } }
I20260811 02:41:24.777597 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.775278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.2.251.1:45263)
0811 02:41:24.775342 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:24.775344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:24.775354 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:41:24.775402 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:24.775403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:24.775427 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:24.775490 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:41:24.775612 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.776095 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.776862 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:24.777051 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:24.777107 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:24.777259 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:41:24.777374 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:41:24.777449 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:24.777489 (+ 40us) server_negotiation.cc:299] Negotiation successful
0811 02:41:24.777513 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:41:24.779662 32704 raft_consensus.cc:1261] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 1 FOLLOWER]: Refusing update from remote peer 6f720b03a8b94c91be158ad0151d9259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:24.779661 32493 raft_consensus.cc:1261] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 1 FOLLOWER]: Refusing update from remote peer 6f720b03a8b94c91be158ad0151d9259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:24.780133 343 consensus_queue.cc:1035] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f69c9cafdf04744808ccc98e91e9282" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:24.780287 336 consensus_queue.cc:1035] T e9eb6721d4854f2684b6ae21d287d981 P 6f720b03a8b94c91be158ad0151d9259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be7d1f019e184ddcace6a21f9ec7391d" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:24.783334 348 mvcc.cc:205] Tried to move back new op lower bound from 7317160283254329344 to 7317160283034525696. Current Snapshot: MvccSnapshot[applied={T|T < 7317160283254329344}]
I20260811 02:41:24.783651 347 mvcc.cc:205] Tried to move back new op lower bound from 7317160283254329344 to 7317160283034525696. Current Snapshot: MvccSnapshot[applied={T|T < 7317160283254329344 or (T in {7317160283254329344})}]
I20260811 02:41:24.785012 32021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:24.785107 32021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:24.785917 32021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c63b542ac304632be1ceb030bc83a14: Sending DeleteTablet for 3 replicas of tablet e9eb6721d4854f2684b6ae21d287d981
I20260811 02:41:24.786353 32454 tablet_service.cc:1514] Processing DeleteTablet for tablet e9eb6721d4854f2684b6ae21d287d981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:24 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:32864
I20260811 02:41:24.786612 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 32122
I20260811 02:41:24.786583 32230 tablet_service.cc:1514] Processing DeleteTablet for tablet e9eb6721d4854f2684b6ae21d287d981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:24 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:43704
I20260811 02:41:24.787156 32678 tablet_service.cc:1514] Processing DeleteTablet for tablet e9eb6721d4854f2684b6ae21d287d981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:24 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:59654
I20260811 02:41:24.789600 362 tablet_replica.cc:330] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: stopping tablet replica
I20260811 02:41:24.789789 362 raft_consensus.cc:2229] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:24.789965 362 raft_consensus.cc:2258] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:24.794642 361 tablet_replica.cc:330] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: stopping tablet replica
I20260811 02:41:24.794852 361 raft_consensus.cc:2229] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:24.795028 361 raft_consensus.cc:2258] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:24.799212 361 ts_tablet_manager.cc:1905] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:24.801501 362 ts_tablet_manager.cc:1905] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:24.801733 361 ts_tablet_manager.cc:1918] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:24.801895 361 log.cc:1196] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/e9eb6721d4854f2684b6ae21d287d981
I20260811 02:41:24.802170 361 ts_tablet_manager.cc:1939] T e9eb6721d4854f2684b6ae21d287d981 P 9f69c9cafdf04744808ccc98e91e9282: Deleting consensus metadata
I20260811 02:41:24.802691 31995 catalog_manager.cc:4841] TS 9f69c9cafdf04744808ccc98e91e9282 (127.2.251.2:37391): tablet e9eb6721d4854f2684b6ae21d287d981 (table test-table [id=87f61be9a44447788ae9891550e5dfe8]) successfully deleted
I20260811 02:41:24.804045 362 ts_tablet_manager.cc:1918] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:24.804126 362 log.cc:1196] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/e9eb6721d4854f2684b6ae21d287d981
I20260811 02:41:24.804345 362 ts_tablet_manager.cc:1939] T e9eb6721d4854f2684b6ae21d287d981 P be7d1f019e184ddcace6a21f9ec7391d: Deleting consensus metadata
I20260811 02:41:24.804885 31987 catalog_manager.cc:4841] TS be7d1f019e184ddcace6a21f9ec7391d (127.2.251.3:46143): tablet e9eb6721d4854f2684b6ae21d287d981 (table test-table [id=87f61be9a44447788ae9891550e5dfe8]) successfully deleted
W20260811 02:41:24.809607 31988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:45263 (error 108)
I20260811 02:41:24.810166 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 32350
I20260811 02:41:24.810204 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:24.810086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:45263 (local address 127.0.0.1:43720)
0811 02:41:24.810137 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:24.810172 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:41:24.825030 31988 catalog_manager.cc:4584] TS 6f720b03a8b94c91be158ad0151d9259 (127.2.251.1:45263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9eb6721d4854f2684b6ae21d287d981: Network error: Client connection negotiation failed: client connection to 127.2.251.1:45263: connect: Connection refused (error 111)
I20260811 02:41:24.842406 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 32575
I20260811 02:41:24.856303 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 31955
2026-08-11T02:41:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2488 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T02:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:24Z Disabled control of system clock
I20260811 02:41:24.899031 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:44631
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:44631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:24.987833 370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:24.988081 370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:24.988103 370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:24.989355 370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:24.989408 370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:24.989419 370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:24.989431 370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:24.989444 370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:24.990926 370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:44631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:24.991196 370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:24.991508 370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:24.996573 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:24.996567 381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.996652 370 server_base.cc:1034] running on GCE node
W20260811 02:41:24.996562 384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:24.997133 370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:24.997473 370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:24.998684 370 hybrid_clock.cc:648] HybridClock initialized: now 1786416084998663 us; error 38 us; skew 500 ppm
I20260811 02:41:25.000057 370 webserver.cc:460] Webserver started at http://127.2.251.62:37563/ using document root <none> and password file <none>
I20260811 02:41:25.000315 370 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:25.000365 370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:25.000484 370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:25.001610 370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "4eebb3525a9e42de937b55bcc0380f6e"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "0787fe5037c6ab72770920506797530c"
server_key_iv: "cd9804270d4ca7d1aeca5b4b4d178417"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.002044 370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "4eebb3525a9e42de937b55bcc0380f6e"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "0787fe5037c6ab72770920506797530c"
server_key_iv: "cd9804270d4ca7d1aeca5b4b4d178417"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.003561 370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:25.004813 394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.005096 370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:25.005237 370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "4eebb3525a9e42de937b55bcc0380f6e"
format_stamp: "Formatted at 2026-08-11 02:41:24 on dist-test-slave-cvwc"
server_key: "0787fe5037c6ab72770920506797530c"
server_key_iv: "cd9804270d4ca7d1aeca5b4b4d178417"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.005321 370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:25.032512 370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:25.032884 370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:25.033025 370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:25.037137 370 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:44631
I20260811 02:41:25.037246 472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:44631 every 8 connection(s)
I20260811 02:41:25.037555 370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:25.038661 473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:25.041177 473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: Bootstrap starting.
I20260811 02:41:25.041997 473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:25.042308 473 log.cc:824] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:25.043100 473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: No bootstrap required, opened a new log
I20260811 02:41:25.044795 473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eebb3525a9e42de937b55bcc0380f6e" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44631 } }
I20260811 02:41:25.044961 473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:25.044983 473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eebb3525a9e42de937b55bcc0380f6e, State: Initialized, Role: FOLLOWER
I20260811 02:41:25.045159 473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eebb3525a9e42de937b55bcc0380f6e" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44631 } }
I20260811 02:41:25.045243 473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:25.045269 473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:25.045265 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 370
I20260811 02:41:25.045320 473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:25.045358 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:25.045545 3052 external_mini_cluster.cc:1428] Setting key 2dadd47a1dec81585d230a7a4dbd7926
I20260811 02:41:25.046142 473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eebb3525a9e42de937b55bcc0380f6e" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44631 } }
I20260811 02:41:25.046263 473 leader_election.cc:302] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eebb3525a9e42de937b55bcc0380f6e; no voters:
I20260811 02:41:25.046435 473 leader_election.cc:288] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:25.046485 486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:25.046605 486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [term 1 LEADER]: Becoming Leader. State: Replica: 4eebb3525a9e42de937b55bcc0380f6e, State: Running, Role: LEADER
I20260811 02:41:25.046732 473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:25.046695 486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eebb3525a9e42de937b55bcc0380f6e" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44631 } }
I20260811 02:41:25.047199 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eebb3525a9e42de937b55bcc0380f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eebb3525a9e42de937b55bcc0380f6e" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44631 } } }
I20260811 02:41:25.047319 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:25.047479 489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eebb3525a9e42de937b55bcc0380f6e. Latest consensus state: current_term: 1 leader_uuid: "4eebb3525a9e42de937b55bcc0380f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eebb3525a9e42de937b55bcc0380f6e" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44631 } } }
I20260811 02:41:25.047545 489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e [sys.catalog]: This master's current role is: LEADER
W20260811 02:41:25.048740 510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:41:25.048926 510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:41:25.048978 499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:25.049120 499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:25.050688 499 catalog_manager.cc:1347] Generated new cluster ID: d35ad9acb2ac4ffc95045e76c3d731ec
I20260811 02:41:25.050750 499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:25.052482 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.046100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.2.251.62:44631)
0811 02:41:25.046356 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.046366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.046398 (+ 32us) server_negotiation.cc:407] Connection header received
0811 02:41:25.046522 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.046530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.046613 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.047026 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:25.047134 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.049217 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.050342 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.050938 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.051059 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.051741 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:25.051745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:25.051745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:25.051942 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:25.051978 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:25.051993 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:25.052047 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.052137 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.052248 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
I20260811 02:41:25.056584 499 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:25.057108 499 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:25.067854 499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: Generated new TSK 0
I20260811 02:41:25.068079 499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:25.077852 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44631
--builtin_ntp_servers=127.2.251.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:25.169243 514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:25.169481 514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:25.169507 514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:25.170758 514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:25.170809 514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:25.170835 514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:25.172379 514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:25.172642 514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:25.172948 514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:25.178437 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:25.178478 521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:25.178597 514 server_base.cc:1034] running on GCE node
W20260811 02:41:25.178454 528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:25.178892 514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:25.179142 514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:25.180326 514 hybrid_clock.cc:648] HybridClock initialized: now 1786416085180274 us; error 63 us; skew 500 ppm
I20260811 02:41:25.181794 514 webserver.cc:460] Webserver started at http://127.2.251.1:35845/ using document root <none> and password file <none>
I20260811 02:41:25.182075 514 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:25.182130 514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:25.182264 514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:25.183494 514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "7a56a074735f47e4ae69bb7af908688c"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "3198f22b9d5fa33269275280e4d737b3"
server_key_iv: "fba3378053c1282684d6bc6495d45e7b"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.183843 514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "7a56a074735f47e4ae69bb7af908688c"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "3198f22b9d5fa33269275280e4d737b3"
server_key_iv: "fba3378053c1282684d6bc6495d45e7b"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.185281 514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:41:25.186474 543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.186738 514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:25.186851 514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "7a56a074735f47e4ae69bb7af908688c"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "3198f22b9d5fa33269275280e4d737b3"
server_key_iv: "fba3378053c1282684d6bc6495d45e7b"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.186919 514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:25.195036 514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:25.195353 514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:25.195478 514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:25.195744 514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:25.196146 514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:25.196185 514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.196214 514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:25.196238 514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.202716 514 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:44105
I20260811 02:41:25.202759 722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:44105 every 8 connection(s)
I20260811 02:41:25.203200 514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:41:25.204080 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 514
I20260811 02:41:25.204170 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:41:25.204409 3052 external_mini_cluster.cc:1428] Setting key 1bb2d801b7758918430d78aacefd1d99
I20260811 02:41:25.205581 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44631
--builtin_ntp_servers=127.2.251.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 02:41:25.209137 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.204260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:38687 (local address 127.2.251.62:44631)
0811 02:41:25.204314 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.204316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.204565 (+ 249us) server_negotiation.cc:407] Connection header received
0811 02:41:25.205204 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.205205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.205219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.205276 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:25.206058 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.206552 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.207611 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.207806 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.207862 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.208864 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:25.208865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:25.208866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:25.208872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:25.208887 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:25.208892 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:25.208936 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.209016 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.209052 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:41:25.209416 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.204216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44631 (local address 127.2.251.1:38687)
0811 02:41:25.204429 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.204474 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.205108 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.205345 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.205349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.205360 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:25.205967 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.205973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.206650 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.206651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.207525 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.207528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.207878 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.207879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.208024 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.208709 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:25.208709 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:25.208740 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:25.208776 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:25.208943 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:25.208949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:25.208953 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.209016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.209130 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260811 02:41:25.209916 723 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44631
I20260811 02:41:25.210016 723 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:25.210206 723 heartbeater.cc:511] Master 127.2.251.62:44631 requested a full tablet report, sending...
I20260811 02:41:25.210732 424 ts_manager.cc:194] Registered new tserver with Master: 7a56a074735f47e4ae69bb7af908688c (127.2.251.1:44105)
I20260811 02:41:25.211364 424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.1:38687
W20260811 02:41:25.299435 727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:25.299679 727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:25.299702 727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:25.302235 727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:25.302331 727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:25.302368 727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:25.304183 727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:25.304731 727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:25.305251 727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:25.311256 741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:25.311251 739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:25.311426 727 server_base.cc:1034] running on GCE node
W20260811 02:41:25.311251 738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:25.311789 727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:25.312065 727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:25.313216 727 hybrid_clock.cc:648] HybridClock initialized: now 1786416085313203 us; error 34 us; skew 500 ppm
I20260811 02:41:25.314738 727 webserver.cc:460] Webserver started at http://127.2.251.2:34481/ using document root <none> and password file <none>
I20260811 02:41:25.314996 727 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:25.315050 727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:25.315186 727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:25.316339 727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "77a40c1b14e34a7cb00bed8de5f175a4"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "0f1947118eab9a39082a981bcb8da060"
server_key_iv: "a0d78c69f5a5776ab3ae597e770c40e7"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.316704 727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "77a40c1b14e34a7cb00bed8de5f175a4"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "0f1947118eab9a39082a981bcb8da060"
server_key_iv: "a0d78c69f5a5776ab3ae597e770c40e7"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.318168 727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:41:25.319070 755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.319240 727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:25.319342 727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "77a40c1b14e34a7cb00bed8de5f175a4"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "0f1947118eab9a39082a981bcb8da060"
server_key_iv: "a0d78c69f5a5776ab3ae597e770c40e7"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.319412 727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:25.338706 727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:25.339093 727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:25.339257 727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:25.339514 727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:25.339908 727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:25.339957 727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.339988 727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:25.340004 727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.347116 727 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:44609
I20260811 02:41:25.347158 928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:44609 every 8 connection(s)
I20260811 02:41:25.347530 727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:25.353012 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.348347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:46391 (local address 127.2.251.62:44631)
0811 02:41:25.348394 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.348396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.348628 (+ 232us) server_negotiation.cc:407] Connection header received
0811 02:41:25.349273 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.349274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.349288 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.349340 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:25.350073 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.350583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.351561 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.351761 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.351820 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.352715 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:25.352717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:25.352717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:25.352723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:25.352738 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:25.352743 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:25.352787 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.352883 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.352920 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:41:25.353220 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.348335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44631 (local address 127.2.251.2:46391)
0811 02:41:25.348505 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.348539 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.349156 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.349410 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.349413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.349422 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:25.349983 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.349988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.350676 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.350677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.351474 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.351477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.351845 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.351846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.351975 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.352559 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:25.352559 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:25.352590 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:25.352616 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:25.352815 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:25.352820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:25.352822 (+ 2us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.352881 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.352994 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260811 02:41:25.353478 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 727
I20260811 02:41:25.353578 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:41:25.353693 929 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44631
I20260811 02:41:25.353823 929 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:25.353832 3052 external_mini_cluster.cc:1428] Setting key 25336d3ba481b0132200b231e1a78a4a
I20260811 02:41:25.354050 929 heartbeater.cc:511] Master 127.2.251.62:44631 requested a full tablet report, sending...
I20260811 02:41:25.354421 422 ts_manager.cc:194] Registered new tserver with Master: 77a40c1b14e34a7cb00bed8de5f175a4 (127.2.251.2:44609)
I20260811 02:41:25.354773 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44631
--builtin_ntp_servers=127.2.251.20:36949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 02:41:25.354903 422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.2:46391
W20260811 02:41:25.441015 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:25.441254 933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:25.441287 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:25.442490 933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:25.442534 933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:25.442571 933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:25.443964 933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:25.444208 933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:25.444505 933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:25.449993 947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:25.450001 944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:25.450030 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:25.450263 933 server_base.cc:1034] running on GCE node
I20260811 02:41:25.450505 933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:25.450760 933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:25.451933 933 hybrid_clock.cc:648] HybridClock initialized: now 1786416085451908 us; error 34 us; skew 500 ppm
I20260811 02:41:25.453469 933 webserver.cc:460] Webserver started at http://127.2.251.3:33581/ using document root <none> and password file <none>
I20260811 02:41:25.453754 933 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:25.453820 933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:25.453972 933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:25.455076 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "fd21f01d73114ec9b26ad8b8c54332b9"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "8d76b20d269190ab2b5ded6d9c3b904a"
server_key_iv: "66d099c0d58b6af75c0ace26d9991cb7"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.455438 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "fd21f01d73114ec9b26ad8b8c54332b9"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "8d76b20d269190ab2b5ded6d9c3b904a"
server_key_iv: "66d099c0d58b6af75c0ace26d9991cb7"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.456849 933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:25.457684 956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.457870 933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:25.457978 933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "fd21f01d73114ec9b26ad8b8c54332b9"
format_stamp: "Formatted at 2026-08-11 02:41:25 on dist-test-slave-cvwc"
server_key: "8d76b20d269190ab2b5ded6d9c3b904a"
server_key_iv: "66d099c0d58b6af75c0ace26d9991cb7"
server_key_version: "encryptionkey@0"
I20260811 02:41:25.458055 933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:25.468808 933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:25.469132 933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:25.469265 933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:25.469482 933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:25.469916 933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:25.469955 933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.469986 933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:25.470005 933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:25.476166 933 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:45089
I20260811 02:41:25.476220 1141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:45089 every 8 connection(s)
I20260811 02:41:25.476555 933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:25.481873 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.477283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:44799 (local address 127.2.251.62:44631)
0811 02:41:25.477416 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.477419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.477645 (+ 226us) server_negotiation.cc:407] Connection header received
0811 02:41:25.478139 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.478141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.478153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.478232 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:25.478917 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.479436 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.480461 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.480666 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.480728 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.481586 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:25.481587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:25.481587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:25.481592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:25.481607 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:25.481611 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:25.481662 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.481782 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.481810 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 02:41:25.481936 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 933
I20260811 02:41:25.482030 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:41:25.482141 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.477253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44631 (local address 127.2.251.3:44799)
0811 02:41:25.477479 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.477522 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.478052 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.478288 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.478291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.478300 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:25.478827 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.478831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.479542 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.479544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.480354 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.480357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.480752 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.480753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.480872 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.481435 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:25.481435 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:25.481465 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:25.481491 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:25.481688 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:25.481692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:25.481696 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.481786 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.481906 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260811 02:41:25.482223 3052 external_mini_cluster.cc:1428] Setting key a75c98270cbbba810177c747b611ba60
I20260811 02:41:25.482658 1142 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44631
I20260811 02:41:25.482761 1142 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:25.482949 1142 heartbeater.cc:511] Master 127.2.251.62:44631 requested a full tablet report, sending...
I20260811 02:41:25.483294 422 ts_manager.cc:194] Registered new tserver with Master: fd21f01d73114ec9b26ad8b8c54332b9 (127.2.251.3:45089)
I20260811 02:41:25.483659 422 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.3:44799
I20260811 02:41:25.493963 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:41:25.498191 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.495379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.2.251.62:44631)
0811 02:41:25.495456 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.495458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.495536 (+ 78us) server_negotiation.cc:407] Connection header received
0811 02:41:25.495586 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.495587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.495600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.495649 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:25.495780 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.496260 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.497031 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.497212 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.497263 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.497936 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:25.497938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:25.497938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:25.497942 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:25.497956 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:25.497960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:25.498007 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.498091 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.498117 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 02:41:25.500384 422 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:41:25.502017 422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:41:25.508520 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.504623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:44609 (local address 127.0.0.1:47230)
0811 02:41:25.504998 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.505009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.505087 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.505350 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.505353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.505357 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:25.505437 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.505439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.506389 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.506389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.507321 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.507323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.508199 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.508200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.508282 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.508297 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.508378 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.508434 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":133,"threads_started":1}
I20260811 02:41:25.508533 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.504830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.2.251.2:44609)
0811 02:41:25.505021 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.505027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.505059 (+ 32us) server_negotiation.cc:407] Connection header received
0811 02:41:25.505223 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.505226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.505243 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.505334 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:25.505497 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.506310 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.507428 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.508127 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.508199 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.508294 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.508392 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.508434 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260811 02:41:25.509145 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.504635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.2.251.1:44105)
0811 02:41:25.505206 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.505210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.505219 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:41:25.505258 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.505260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.505274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.505526 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:25.505822 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.507023 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.508082 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.508792 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.508861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.508955 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.509027 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.509065 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":96,"threads_started":1}
I20260811 02:41:25.509138 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.504413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:44105 (local address 127.0.0.1:36296)
0811 02:41:25.504682 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.504727 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.504797 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.505538 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.505539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.505542 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:25.505595 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.505597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.507159 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.507161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.508003 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.508005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.508877 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.508878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.508953 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.508957 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.509012 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.509057 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260811 02:41:25.509696 822 tablet_service.cc:1467] Processing CreateTablet for tablet bdbc6a52b8da4daba177cd1677f735a3 (DEFAULT_TABLE table=test-table [id=2732b2f46c0d4d9d877260007d5056ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:25.510157 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbc6a52b8da4daba177cd1677f735a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:25.510205 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.504635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:45089 (local address 127.0.0.1:58506)
0811 02:41:25.505382 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.505390 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.505435 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.505665 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.505666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.505668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:25.505706 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.505707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.508080 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.508081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.509078 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.509082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.509970 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.509972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.510036 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.510040 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.510093 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.510134 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":97,"threads_started":1}
I20260811 02:41:25.510231 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.504885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.2.251.3:45089)
0811 02:41:25.505072 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.505077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.505514 (+ 437us) server_negotiation.cc:407] Connection header received
0811 02:41:25.505571 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.505573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.505588 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.505656 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:25.505794 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.507022 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.509169 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.509843 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.509906 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.509997 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.510109 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.510150 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260811 02:41:25.510812 1039 tablet_service.cc:1467] Processing CreateTablet for tablet bdbc6a52b8da4daba177cd1677f735a3 (DEFAULT_TABLE table=test-table [id=2732b2f46c0d4d9d877260007d5056ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:25.511116 1039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbc6a52b8da4daba177cd1677f735a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:25.512974 1161 tablet_bootstrap.cc:492] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: Bootstrap starting.
I20260811 02:41:25.513510 616 tablet_service.cc:1467] Processing CreateTablet for tablet bdbc6a52b8da4daba177cd1677f735a3 (DEFAULT_TABLE table=test-table [id=2732b2f46c0d4d9d877260007d5056ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:25.513675 1161 tablet_bootstrap.cc:654] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:25.514031 1162 tablet_bootstrap.cc:492] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Bootstrap starting.
I20260811 02:41:25.514052 1161 log.cc:824] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:25.514108 616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbc6a52b8da4daba177cd1677f735a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:25.514662 1162 tablet_bootstrap.cc:654] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:25.514896 1161 tablet_bootstrap.cc:492] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: No bootstrap required, opened a new log
I20260811 02:41:25.514976 1161 ts_tablet_manager.cc:1397] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:41:25.514945 1162 log.cc:824] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:25.515621 1162 tablet_bootstrap.cc:492] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: No bootstrap required, opened a new log
I20260811 02:41:25.515681 1162 ts_tablet_manager.cc:1397] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:41:25.517495 1162 raft_consensus.cc:348] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.517692 1162 raft_consensus.cc:374] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:25.517771 1162 raft_consensus.cc:729] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd21f01d73114ec9b26ad8b8c54332b9, State: Initialized, Role: FOLLOWER
I20260811 02:41:25.517704 1161 raft_consensus.cc:348] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.517906 1161 raft_consensus.cc:374] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:25.517946 1161 raft_consensus.cc:729] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a40c1b14e34a7cb00bed8de5f175a4, State: Initialized, Role: FOLLOWER
I20260811 02:41:25.517927 1162 consensus_queue.cc:260] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.517660 1167 tablet_bootstrap.cc:492] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: Bootstrap starting.
I20260811 02:41:25.518052 1161 consensus_queue.cc:260] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.518220 1162 ts_tablet_manager.cc:1428] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:41:25.518299 1161 ts_tablet_manager.cc:1428] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 02:41:25.518350 1142 heartbeater.cc:503] Master 127.2.251.62:44631 was elected leader, sending a full tablet report...
I20260811 02:41:25.518414 929 heartbeater.cc:503] Master 127.2.251.62:44631 was elected leader, sending a full tablet report...
I20260811 02:41:25.518827 1167 tablet_bootstrap.cc:654] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:25.519166 1167 log.cc:824] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:25.520001 1167 tablet_bootstrap.cc:492] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: No bootstrap required, opened a new log
I20260811 02:41:25.520069 1167 ts_tablet_manager.cc:1397] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 02:41:25.522018 1167 raft_consensus.cc:348] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.522173 1167 raft_consensus.cc:374] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:25.522197 1167 raft_consensus.cc:729] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a56a074735f47e4ae69bb7af908688c, State: Initialized, Role: FOLLOWER
I20260811 02:41:25.522310 1167 consensus_queue.cc:260] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.522564 1167 ts_tablet_manager.cc:1428] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:41:25.522660 723 heartbeater.cc:503] Master 127.2.251.62:44631 was elected leader, sending a full tablet report...
W20260811 02:41:25.598170 931 tablet.cc:2369] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:25.685146 1171 raft_consensus.cc:482] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:25.685313 1171 raft_consensus.cc:504] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.685684 1171 leader_election.cc:288] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a56a074735f47e4ae69bb7af908688c (127.2.251.1:44105), fd21f01d73114ec9b26ad8b8c54332b9 (127.2.251.3:45089)
I20260811 02:41:25.689538 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.685931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:44105 (local address 127.2.251.2:39219)
0811 02:41:25.685987 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.686000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.686066 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.686227 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.686228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.686230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:25.686321 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.686323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.687214 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.687215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.688518 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.688522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.689252 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.689253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.689339 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.689345 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.689405 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.689450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":32}
I20260811 02:41:25.689543 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.686132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:51089 (local address 127.2.251.3:45089)
0811 02:41:25.686178 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.686180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.686245 (+ 65us) server_negotiation.cc:407] Connection header received
0811 02:41:25.686351 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.686352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.686362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.686417 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:25.686529 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.687210 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.688606 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.689160 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.689229 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.689240 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.689415 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.689470 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:41:25.689543 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.686026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:39219 (local address 127.2.251.1:44105)
0811 02:41:25.686082 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.686084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.686095 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:41:25.686129 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.686130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.686143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.686200 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:25.686400 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.687131 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.688607 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.689164 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.689229 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.689239 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.689415 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.689469 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:41:25.689544 1179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.686028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:45089 (local address 127.2.251.2:51089)
0811 02:41:25.686193 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.686201 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:25.686277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:25.686428 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:25.686429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:25.686430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:25.686478 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.686479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.687276 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.687277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.688522 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:25.688523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.689246 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.689248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:25.689339 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.689346 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 02:41:25.689404 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 02:41:25.689449 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":29,"thread_start_us":115,"threads_started":1}
I20260811 02:41:25.690016 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbc6a52b8da4daba177cd1677f735a3" candidate_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" is_pre_election: true
I20260811 02:41:25.690003 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbc6a52b8da4daba177cd1677f735a3" candidate_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a56a074735f47e4ae69bb7af908688c" is_pre_election: true
I20260811 02:41:25.690187 648 raft_consensus.cc:2454] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a40c1b14e34a7cb00bed8de5f175a4 in term 0.
I20260811 02:41:25.690187 1068 raft_consensus.cc:2454] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a40c1b14e34a7cb00bed8de5f175a4 in term 0.
I20260811 02:41:25.690449 757 leader_election.cc:302] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a40c1b14e34a7cb00bed8de5f175a4, fd21f01d73114ec9b26ad8b8c54332b9; no voters:
I20260811 02:41:25.690625 1171 raft_consensus.cc:2781] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:25.690688 1171 raft_consensus.cc:482] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:25.690704 1171 raft_consensus.cc:3037] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:25.691443 1171 raft_consensus.cc:504] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.691555 1171 leader_election.cc:288] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [CANDIDATE]: Term 1 election: Requested vote from peers 7a56a074735f47e4ae69bb7af908688c (127.2.251.1:44105), fd21f01d73114ec9b26ad8b8c54332b9 (127.2.251.3:45089)
I20260811 02:41:25.691746 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbc6a52b8da4daba177cd1677f735a3" candidate_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a56a074735f47e4ae69bb7af908688c"
I20260811 02:41:25.691763 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbc6a52b8da4daba177cd1677f735a3" candidate_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd21f01d73114ec9b26ad8b8c54332b9"
I20260811 02:41:25.691839 1068 raft_consensus.cc:3037] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:25.691839 648 raft_consensus.cc:3037] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:25.692674 1068 raft_consensus.cc:2454] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a40c1b14e34a7cb00bed8de5f175a4 in term 1.
I20260811 02:41:25.692674 648 raft_consensus.cc:2454] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a40c1b14e34a7cb00bed8de5f175a4 in term 1.
I20260811 02:41:25.692875 757 leader_election.cc:302] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a40c1b14e34a7cb00bed8de5f175a4, 7a56a074735f47e4ae69bb7af908688c; no voters:
I20260811 02:41:25.692960 1171 raft_consensus.cc:2781] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:25.693092 1171 raft_consensus.cc:686] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 1 LEADER]: Becoming Leader. State: Replica: 77a40c1b14e34a7cb00bed8de5f175a4, State: Running, Role: LEADER
I20260811 02:41:25.693169 1171 consensus_queue.cc:237] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } }
I20260811 02:41:25.694087 422 catalog_manager.cc:5495] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77a40c1b14e34a7cb00bed8de5f175a4 (127.2.251.2). New cstate: current_term: 1 leader_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a40c1b14e34a7cb00bed8de5f175a4" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44609 } health_report { overall_health: HEALTHY } } }
I20260811 02:41:25.698803 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.696566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.2.251.2:44609)
0811 02:41:25.696603 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:25.696605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:25.696627 (+ 22us) server_negotiation.cc:407] Connection header received
0811 02:41:25.696697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:25.696698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:25.696718 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:25.696778 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:41:25.696886 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.697338 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.698099 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:25.698292 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:25.698345 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:25.698499 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:41:25.698599 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:41:25.698671 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:25.698691 (+ 20us) server_negotiation.cc:299] Negotiation successful
0811 02:41:25.698713 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:41:25.700639 1068 raft_consensus.cc:1261] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 1 FOLLOWER]: Refusing update from remote peer 77a40c1b14e34a7cb00bed8de5f175a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:25.700991 1171 consensus_queue.cc:1035] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd21f01d73114ec9b26ad8b8c54332b9" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:25.701073 648 raft_consensus.cc:1261] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c [term 1 FOLLOWER]: Refusing update from remote peer 77a40c1b14e34a7cb00bed8de5f175a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:25.701442 1171 consensus_queue.cc:1035] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a56a074735f47e4ae69bb7af908688c" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 02:41:25.704059 725 tablet.cc:2369] T bdbc6a52b8da4daba177cd1677f735a3 P 7a56a074735f47e4ae69bb7af908688c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:25.706666 1186 mvcc.cc:205] Tried to move back new op lower bound from 7317160287026765824 to 7317160286999744512. Current Snapshot: MvccSnapshot[applied={T|T < 7317160287026765824 or (T in {7317160287026765824})}]
I20260811 02:41:25.706948 424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:25.707039 424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:25.707831 424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4eebb3525a9e42de937b55bcc0380f6e: Sending DeleteTablet for 3 replicas of tablet bdbc6a52b8da4daba177cd1677f735a3
I20260811 02:41:25.708199 822 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbc6a52b8da4daba177cd1677f735a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:25 UTC) from {username='slave'} at 127.0.0.1:47230
I20260811 02:41:25.708201 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbc6a52b8da4daba177cd1677f735a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:25 UTC) from {username='slave'} at 127.0.0.1:58506
I20260811 02:41:25.708446 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 514
I20260811 02:41:25.708506 616 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbc6a52b8da4daba177cd1677f735a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:25 UTC) from {username='slave'} at 127.0.0.1:36296
W20260811 02:41:25.726024 397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:44105 (error 108)
I20260811 02:41:25.726130 1202 tablet_replica.cc:330] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: stopping tablet replica
I20260811 02:41:25.726279 1202 raft_consensus.cc:2229] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:25.726425 1202 raft_consensus.cc:2258] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:25.726629 1203 tablet_replica.cc:330] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4: stopping tablet replica
I20260811 02:41:25.726739 1203 raft_consensus.cc:2229] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:41:25.726748 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.726592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:44105 (local address 127.0.0.1:36304)
0811 02:41:25.726653 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.726707 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 02:41:25.726847 1203 raft_consensus.cc:2258] T bdbc6a52b8da4daba177cd1677f735a3 P 77a40c1b14e34a7cb00bed8de5f175a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:25.726892 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 727
I20260811 02:41:25.727419 1202 ts_tablet_manager.cc:1905] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:25.729573 1202 ts_tablet_manager.cc:1918] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:25.729734 1202 log.cc:1196] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/bdbc6a52b8da4daba177cd1677f735a3
I20260811 02:41:25.730022 1202 ts_tablet_manager.cc:1939] T bdbc6a52b8da4daba177cd1677f735a3 P fd21f01d73114ec9b26ad8b8c54332b9: Deleting consensus metadata
W20260811 02:41:25.741621 397 catalog_manager.cc:4584] TS 7a56a074735f47e4ae69bb7af908688c (127.2.251.1:44105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdbc6a52b8da4daba177cd1677f735a3: Network error: Client connection negotiation failed: client connection to 127.2.251.1:44105: connect: Connection refused (error 111)
I20260811 02:41:25.742887 397 catalog_manager.cc:4841] TS fd21f01d73114ec9b26ad8b8c54332b9 (127.2.251.3:45089): tablet bdbc6a52b8da4daba177cd1677f735a3 (table test-table [id=2732b2f46c0d4d9d877260007d5056ec]) successfully deleted
I20260811 02:41:25.748806 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 933
I20260811 02:41:25.748885 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:25.748770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:44609 (local address 127.0.0.1:47242)
0811 02:41:25.748822 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:25.748851 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.2:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260811 02:41:25.759799 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 370
2026-08-11T02:41:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (900 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:25 dist-test-slave-cvwc krb5kdc[1209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:25 dist-test-slave-cvwc krb5kdc[1209](info): set up 2 sockets
Aug 11 02:41:25 dist-test-slave-cvwc krb5kdc[1209](info): commencing operation
krb5kdc: starting...
W20260811 02:41:26.959517 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:27.092054 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:45115
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:45115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:27.178674 1229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:27.178961 1229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:27.179000 1229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:27.180347 1229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:27.180401 1229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:27.180421 1229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:27.180437 1229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:27.180496 1229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:27.182417 1229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:45115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:45115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:27.182934 1229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:27.183183 1229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:27.188854 1235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.188810 1238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.188881 1236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:27.189373 1229 server_base.cc:1034] running on GCE node
I20260811 02:41:27.189848 1229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:27.190188 1229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:27.191387 1229 hybrid_clock.cc:648] HybridClock initialized: now 1786416087191340 us; error 61 us; skew 500 ppm
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:27.194458 1229 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:27.194983 1229 webserver.cc:460] Webserver started at http://127.2.251.62:43967/ using document root <none> and password file <none>
I20260811 02:41:27.195199 1229 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:27.195250 1229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:27.195395 1229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:27.196503 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "5388382f377c467bbe9ca7d0436e3346"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "f0adfc1e59f668c446e2ed6e5694ef11"
server_key_iv: "3bc69cb596d725329beeef3831cf7b3c"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.196868 1229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "5388382f377c467bbe9ca7d0436e3346"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "f0adfc1e59f668c446e2ed6e5694ef11"
server_key_iv: "3bc69cb596d725329beeef3831cf7b3c"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.198309 1229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:27.199158 1245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.199487 1229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:27.199591 1229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "5388382f377c467bbe9ca7d0436e3346"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "f0adfc1e59f668c446e2ed6e5694ef11"
server_key_iv: "3bc69cb596d725329beeef3831cf7b3c"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.199663 1229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:27.208921 1229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:27.211206 1229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:27.211369 1229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:27.215740 1229 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:45115
I20260811 02:41:27.215791 1341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:45115 every 8 connection(s)
I20260811 02:41:27.216208 1229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:27.217222 1342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:27.218586 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 1229
I20260811 02:41:27.218730 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:27.218940 3052 external_mini_cluster.cc:1428] Setting key da87d63473dc42ee6cc8c7447cbec53b
I20260811 02:41:27.220098 1342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346: Bootstrap starting.
I20260811 02:41:27.220690 1342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:27.220961 1342 log.cc:824] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:27.221763 1342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346: No bootstrap required, opened a new log
I20260811 02:41:27.223594 1342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5388382f377c467bbe9ca7d0436e3346" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 45115 } }
I20260811 02:41:27.223752 1342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:27.223790 1342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388382f377c467bbe9ca7d0436e3346, State: Initialized, Role: FOLLOWER
I20260811 02:41:27.223942 1342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5388382f377c467bbe9ca7d0436e3346" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 45115 } }
I20260811 02:41:27.224023 1342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:27.224057 1342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:27.224107 1342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:27.224709 1342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5388382f377c467bbe9ca7d0436e3346" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 45115 } }
I20260811 02:41:27.224825 1342 leader_election.cc:302] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5388382f377c467bbe9ca7d0436e3346; no voters:
I20260811 02:41:27.225004 1342 leader_election.cc:288] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:27.225070 1348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:27.225230 1348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [term 1 LEADER]: Becoming Leader. State: Replica: 5388382f377c467bbe9ca7d0436e3346, State: Running, Role: LEADER
I20260811 02:41:27.225291 1342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:27.225358 1348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5388382f377c467bbe9ca7d0436e3346" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 45115 } }
I20260811 02:41:27.225792 1348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5388382f377c467bbe9ca7d0436e3346. Latest consensus state: current_term: 1 leader_uuid: "5388382f377c467bbe9ca7d0436e3346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5388382f377c467bbe9ca7d0436e3346" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 45115 } } }
I20260811 02:41:27.225878 1348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:27.225838 1352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5388382f377c467bbe9ca7d0436e3346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5388382f377c467bbe9ca7d0436e3346" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 45115 } } }
I20260811 02:41:27.225920 1352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:27.226092 1364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:27.226922 1364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:27.229104 1364 catalog_manager.cc:1347] Generated new cluster ID: 27d8047d7f42417a9b1678f87fd0159e
I20260811 02:41:27.229156 1364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:27.231335 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.219554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.2.251.62:45115)
0811 02:41:27.219740 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.219748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.219782 (+ 34us) server_negotiation.cc:407] Connection header received
0811 02:41:27.220207 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.220216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.220281 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.220413 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:27.221102 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.222488 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.223472 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.224059 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.224320 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.227343 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:27.227348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:27.227349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:27.227364 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:27.229519 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.230010 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.230014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.230014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.230139 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.230325 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.230326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.230326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.230679 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:27.230866 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.230941 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.231103 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260811 02:41:27.235405 1364 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:27.235965 1364 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:27.247763 1364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346: Generated new TSK 0
I20260811 02:41:27.248034 1364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:41:27.285311 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:45115
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:27.373507 1387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:27.373752 1387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:27.373778 1387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:27.374987 1387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:27.375034 1387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:27.375061 1387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:27.376603 1387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:27.376874 1387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:27.377139 1387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:27.382447 1395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.382447 1396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.382470 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:27.382826 1387 server_base.cc:1034] running on GCE node
I20260811 02:41:27.383100 1387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:27.383335 1387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:27.384493 1387 hybrid_clock.cc:648] HybridClock initialized: now 1786416087384481 us; error 34 us; skew 500 ppm
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:27.391867 1387 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:41:27.392352 1387 webserver.cc:460] Webserver started at http://127.2.251.1:36737/ using document root <none> and password file <none>
I20260811 02:41:27.392602 1387 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:27.392664 1387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:27.392788 1387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:27.393931 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "a8920722b82c4d0a935c12b227a4de03"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "63f598b19ced20d98bb979e3e1d767ca"
server_key_iv: "7d5fe4c05804a23c67836a59a91fa62c"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.394254 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "a8920722b82c4d0a935c12b227a4de03"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "63f598b19ced20d98bb979e3e1d767ca"
server_key_iv: "7d5fe4c05804a23c67836a59a91fa62c"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.395646 1387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:27.396502 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.396678 1387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:27.396768 1387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "a8920722b82c4d0a935c12b227a4de03"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "63f598b19ced20d98bb979e3e1d767ca"
server_key_iv: "7d5fe4c05804a23c67836a59a91fa62c"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.396833 1387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:27.404696 1387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:27.406965 1387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:27.407125 1387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:27.407379 1387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:27.407721 1387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:27.407760 1387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.407792 1387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:27.407810 1387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.414469 1387 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:32797
I20260811 02:41:27.414557 1586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:32797 every 8 connection(s)
I20260811 02:41:27.414904 1387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:27.422278 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 1387
I20260811 02:41:27.422391 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:41:27.422607 3052 external_mini_cluster.cc:1428] Setting key 49dfb29bb6c70af3a19353c9cbfd4de0
I20260811 02:41:27.424891 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.415737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:58155 (local address 127.2.251.62:45115)
0811 02:41:27.415809 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.415811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.415979 (+ 168us) server_negotiation.cc:407] Connection header received
0811 02:41:27.416525 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.416527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.416539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.416605 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:27.417601 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.418189 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.419236 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.419468 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.419534 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.421769 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:27.421772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:27.421772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:27.421781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:27.423414 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.423933 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.423934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.423934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.423999 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.424268 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.424268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.424268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.424375 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:27.424430 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.424732 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.424813 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:41:27.425124 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.415683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:45115 (local address 127.2.251.1:58155)
0811 02:41:27.415851 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:27.415894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:27.416411 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:27.416669 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:27.416672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:27.417266 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:27.417484 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.417489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.418308 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.418309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.419140 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.419143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.419542 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.419542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.419651 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.420166 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:27.420176 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:27.421628 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:27.423515 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:27.423524 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:27.423527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:27.423825 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:27.424068 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:27.424068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:27.424069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:27.424210 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:27.424425 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:27.424431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:27.424599 (+ 168us) client_negotiation.cc:769] Sending connection context
0811 02:41:27.424710 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 02:41:27.424884 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260811 02:41:27.425770 1587 heartbeater.cc:347] Connected to a master server at 127.2.251.62:45115
I20260811 02:41:27.425868 1587 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:27.426066 1587 heartbeater.cc:511] Master 127.2.251.62:45115 requested a full tablet report, sending...
I20260811 02:41:27.426682 1276 ts_manager.cc:194] Registered new tserver with Master: a8920722b82c4d0a935c12b227a4de03 (127.2.251.1:32797)
I20260811 02:41:27.427549 1276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:58155
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:41:27.454509 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:45115
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:27.546150 1594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:27.546622 1594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:27.546674 1594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:27.548640 1594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:27.548709 1594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:27.548779 1594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:27.551275 1594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:27.551577 1594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:27.551878 1594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:27.562662 1603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.562682 1601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.562678 1600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:27.563246 1594 server_base.cc:1034] running on GCE node
I20260811 02:41:27.563465 1594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:27.563710 1594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:27.564875 1594 hybrid_clock.cc:648] HybridClock initialized: now 1786416087564856 us; error 38 us; skew 500 ppm
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:27.567767 1594 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:41:27.571771 1594 webserver.cc:460] Webserver started at http://127.2.251.2:34483/ using document root <none> and password file <none>
I20260811 02:41:27.572034 1594 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:27.572091 1594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:27.572242 1594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:27.573619 1594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "3e63841c300041c9892fa92c84580ec7"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "06e7bad90f6c9e260852526377ee6805"
server_key_iv: "15d185dfdc67526eca03e23151bef78a"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.574124 1594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "3e63841c300041c9892fa92c84580ec7"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "06e7bad90f6c9e260852526377ee6805"
server_key_iv: "15d185dfdc67526eca03e23151bef78a"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.575846 1594 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 02:41:27.576767 1614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.576953 1594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:27.577055 1594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "3e63841c300041c9892fa92c84580ec7"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "06e7bad90f6c9e260852526377ee6805"
server_key_iv: "15d185dfdc67526eca03e23151bef78a"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.577136 1594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:27.596292 1594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:27.599016 1594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:27.599195 1594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:27.599519 1594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:27.599990 1594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:27.600065 1594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.600101 1594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:27.600119 1594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.608472 1594 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:41073
I20260811 02:41:27.608505 1803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:41073 every 8 connection(s)
I20260811 02:41:27.608971 1594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:27.614008 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 1594
I20260811 02:41:27.614141 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:41:27.614360 3052 external_mini_cluster.cc:1428] Setting key 2ccd90f32546b40c227878495dc4422f
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:27.621383 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.609828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:48375 (local address 127.2.251.62:45115)
0811 02:41:27.609875 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.609878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.610386 (+ 508us) server_negotiation.cc:407] Connection header received
0811 02:41:27.611188 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.611191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.611216 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.611327 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:27.612441 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.612955 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.614164 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.614405 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.614490 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.617149 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:27.617150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:27.617151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:27.617160 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:27.619226 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.619952 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.619953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.619954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.620052 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.620480 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.620483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.620484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.620663 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:27.620744 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.621122 (+ 378us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.621303 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260811 02:41:27.621877 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.609814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:45115 (local address 127.2.251.2:48375)
0811 02:41:27.610118 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:27.610197 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:27.610984 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:27.611418 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:27.611428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:27.612090 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:27.612308 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.612323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.613053 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.613054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.614045 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.614049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.614505 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.614507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.614670 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.615213 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:27.615222 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:27.616993 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:27.619415 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:27.619429 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:27.619433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:27.619809 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:27.620154 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:27.620156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:27.620156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:27.620326 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:27.620751 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:27.620758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:27.620966 (+ 208us) client_negotiation.cc:769] Sending connection context
0811 02:41:27.621098 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 02:41:27.621315 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260811 02:41:27.622723 1806 heartbeater.cc:347] Connected to a master server at 127.2.251.62:45115
I20260811 02:41:27.622851 1806 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:27.623070 1806 heartbeater.cc:511] Master 127.2.251.62:45115 requested a full tablet report, sending...
I20260811 02:41:27.623471 1276 ts_manager.cc:194] Registered new tserver with Master: 3e63841c300041c9892fa92c84580ec7 (127.2.251.2:41073)
I20260811 02:41:27.623850 1276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:48375
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:41:27.645663 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:45115
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:27.734683 1815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:27.734941 1815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:27.734980 1815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:27.736279 1815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:27.736337 1815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:27.736377 1815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:27.738083 1815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:27.738371 1815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:27.738628 1815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:27.744119 1825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.744061 1831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:27.744065 1826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:27.744084 1815 server_base.cc:1034] running on GCE node
I20260811 02:41:27.744586 1815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:27.744856 1815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:27.746007 1815 hybrid_clock.cc:648] HybridClock initialized: now 1786416087745989 us; error 30 us; skew 500 ppm
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:27.749074 1815 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:41:27.749763 1815 webserver.cc:460] Webserver started at http://127.2.251.3:37885/ using document root <none> and password file <none>
I20260811 02:41:27.750011 1815 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:27.750069 1815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:27.750213 1815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:27.751361 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "82fe9ed83f2f40ae89d2f46ad442c339"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "2b267d250357bf9b93281cbc368ee8ae"
server_key_iv: "9a3b344fa5da4b6692dc2165b25cd4a5"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.751727 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "82fe9ed83f2f40ae89d2f46ad442c339"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "2b267d250357bf9b93281cbc368ee8ae"
server_key_iv: "9a3b344fa5da4b6692dc2165b25cd4a5"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.753436 1815 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:41:27.754528 1847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.754882 1815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:27.755015 1815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "82fe9ed83f2f40ae89d2f46ad442c339"
format_stamp: "Formatted at 2026-08-11 02:41:27 on dist-test-slave-cvwc"
server_key: "2b267d250357bf9b93281cbc368ee8ae"
server_key_iv: "9a3b344fa5da4b6692dc2165b25cd4a5"
server_key_version: "encryptionkey@0"
I20260811 02:41:27.755112 1815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:27.766381 1815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:27.768836 1815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:27.769003 1815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:27.769263 1815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:27.769604 1815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:27.769641 1815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.769673 1815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:27.769690 1815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:27.777100 1815 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:43625
I20260811 02:41:27.777278 2023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:43625 every 8 connection(s)
I20260811 02:41:27.777552 1815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:27.782454 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 1815
I20260811 02:41:27.782581 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:41:27.782773 3052 external_mini_cluster.cc:1428] Setting key 010c570f297d95b1b90236961ca4c284
Aug 11 02:41:27 dist-test-slave-cvwc krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416087, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:27.788575 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.778517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:50701 (local address 127.2.251.62:45115)
0811 02:41:27.778632 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.778634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.778823 (+ 189us) server_negotiation.cc:407] Connection header received
0811 02:41:27.779350 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.779352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.779430 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.779541 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:27.780600 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.781194 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.782411 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.782699 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.782786 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.785532 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:27.785535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:27.785535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:27.785547 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:27.787066 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.787550 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.787551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.787551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.787620 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.787942 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.787943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.787944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.788067 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:27.788123 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.788421 (+ 298us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.788502 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 02:41:27.788689 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.778439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:45115 (local address 127.2.251.3:50701)
0811 02:41:27.778638 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:27.778685 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:27.779232 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:27.779601 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:27.779605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:27.780225 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:27.780454 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.780459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.781336 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.781337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.782317 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.782321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.782801 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.782802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.782994 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.783722 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:27.783733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:27.785375 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:27.787171 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:27.787181 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:27.787183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:27.787444 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:27.787716 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:27.787717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:27.787718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:27.787866 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:27.788119 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:27.788122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:27.788255 (+ 133us) client_negotiation.cc:769] Sending connection context
0811 02:41:27.788344 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 02:41:27.788498 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260811 02:41:27.789327 2024 heartbeater.cc:347] Connected to a master server at 127.2.251.62:45115
I20260811 02:41:27.789440 2024 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:27.789646 2024 heartbeater.cc:511] Master 127.2.251.62:45115 requested a full tablet report, sending...
I20260811 02:41:27.790109 1276 ts_manager.cc:194] Registered new tserver with Master: 82fe9ed83f2f40ae89d2f46ad442c339 (127.2.251.3:43625)
I20260811 02:41:27.790499 1276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:50701
I20260811 02:41:27.795492 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:41:27.804194 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.797484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.2.251.62:45115)
0811 02:41:27.797566 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.797568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.797626 (+ 58us) server_negotiation.cc:407] Connection header received
0811 02:41:27.797691 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.797691 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.797701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.797806 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:27.798435 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.798951 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.799717 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.799912 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.799969 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.801403 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:27.801405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:27.801405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:27.801413 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:27.802946 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.803387 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.803388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.803388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.803452 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:27.803645 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:27.803646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:27.803646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:27.803798 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:27.803894 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.804030 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.804094 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 02:41:27.806027 1276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:41:27.807587 1276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:41:27.814726 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.810370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.2.251.2:41073)
0811 02:41:27.810563 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.810568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.810578 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:41:27.810622 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.810625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.810642 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.810728 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:27.811221 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.811964 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.813477 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.814202 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.814274 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.814335 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.814593 (+ 258us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.814639 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260811 02:41:27.814713 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.810109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:32797 (local address 127.0.0.1:52150)
0811 02:41:27.810277 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:27.810327 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:27.810396 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:27.810960 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:27.810961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:27.810965 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:27.811052 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.811055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.812435 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.812436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.813608 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.813610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.814280 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.814281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.814471 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.814479 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 02:41:27.814550 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 02:41:27.814602 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":26,"thread_start_us":51,"threads_started":1}
I20260811 02:41:27.814812 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.810352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49742 (local address 127.2.251.3:43625)
0811 02:41:27.810874 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.810878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.810887 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:41:27.810924 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.810926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.810940 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.811209 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:27.811278 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.812358 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.813420 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.814202 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.814327 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.814666 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.814693 (+ 27us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.814735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":114,"threads_started":1}
I20260811 02:41:27.815001 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.810223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.2.251.1:32797)
0811 02:41:27.810767 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:27.810771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:27.810781 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:41:27.810826 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:27.810828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:27.810842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:27.811130 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:27.811210 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.812313 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.813682 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.814197 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.814265 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.814393 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:27.814846 (+ 453us) server_negotiation.cc:299] Negotiation successful
0811 02:41:27.814915 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":91,"threads_started":1}
I20260811 02:41:27.814725 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.810110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:43625 (local address 127.0.0.1:49742)
0811 02:41:27.810616 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:27.810624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:27.810655 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:27.811097 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:27.811097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:27.811100 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:27.811138 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.811140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.812504 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.812505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.813346 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.813348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.814304 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.814305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.814460 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.814477 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:41:27.814540 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 02:41:27.814604 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":71,"spinlock_wait_cycles":384,"thread_start_us":106,"threads_started":1}
I20260811 02:41:27.815536 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:27.810110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:41073 (local address 127.0.0.1:49620)
0811 02:41:27.810508 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:27.810517 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:27.810554 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:27.810859 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:27.810867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:27.810873 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:27.811016 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.811019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.812063 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.812064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.813345 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:27.813349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:27.814365 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:27.814366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:27.814460 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:27.814477 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:41:27.814549 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 02:41:27.814610 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":28,"thread_start_us":94,"threads_started":1}
I20260811 02:41:27.815793 1481 tablet_service.cc:1467] Processing CreateTablet for tablet 91e4c892b9684ba2a14929bc34653254 (DEFAULT_TABLE table=test-table [id=963c949ebaec44d5ab1848f41c3c159c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:27.816190 1481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e4c892b9684ba2a14929bc34653254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:27.816099 1920 tablet_service.cc:1467] Processing CreateTablet for tablet 91e4c892b9684ba2a14929bc34653254 (DEFAULT_TABLE table=test-table [id=963c949ebaec44d5ab1848f41c3c159c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:27.816140 1688 tablet_service.cc:1467] Processing CreateTablet for tablet 91e4c892b9684ba2a14929bc34653254 (DEFAULT_TABLE table=test-table [id=963c949ebaec44d5ab1848f41c3c159c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:27.816416 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e4c892b9684ba2a14929bc34653254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:27.816422 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e4c892b9684ba2a14929bc34653254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:27.819335 2046 tablet_bootstrap.cc:492] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Bootstrap starting.
I20260811 02:41:27.819933 2050 tablet_bootstrap.cc:492] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Bootstrap starting.
I20260811 02:41:27.820182 2046 tablet_bootstrap.cc:654] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:27.820659 2050 tablet_bootstrap.cc:654] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:27.820777 2046 log.cc:824] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:27.821210 2054 tablet_bootstrap.cc:492] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: Bootstrap starting.
I20260811 02:41:27.821249 2050 log.cc:824] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:27.821676 2046 tablet_bootstrap.cc:492] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: No bootstrap required, opened a new log
I20260811 02:41:27.821779 2046 ts_tablet_manager.cc:1397] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:41:27.821866 2054 tablet_bootstrap.cc:654] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:27.822250 2050 tablet_bootstrap.cc:492] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: No bootstrap required, opened a new log
I20260811 02:41:27.822321 2050 ts_tablet_manager.cc:1397] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:41:27.822429 2054 log.cc:824] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:27.823215 2054 tablet_bootstrap.cc:492] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: No bootstrap required, opened a new log
I20260811 02:41:27.823284 2054 ts_tablet_manager.cc:1397] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 02:41:27.823884 2050 raft_consensus.cc:348] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:27.824028 2050 raft_consensus.cc:374] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:27.824049 2050 raft_consensus.cc:729] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fe9ed83f2f40ae89d2f46ad442c339, State: Initialized, Role: FOLLOWER
I20260811 02:41:27.824148 2050 consensus_queue.cc:260] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:27.824131 2046 raft_consensus.cc:348] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:27.824322 2046 raft_consensus.cc:374] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:27.824358 2046 raft_consensus.cc:729] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e63841c300041c9892fa92c84580ec7, State: Initialized, Role: FOLLOWER
I20260811 02:41:27.824435 2050 ts_tablet_manager.cc:1428] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 02:41:27.824482 2046 consensus_queue.cc:260] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:27.824586 2024 heartbeater.cc:503] Master 127.2.251.62:45115 was elected leader, sending a full tablet report...
I20260811 02:41:27.824769 2046 ts_tablet_manager.cc:1428] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 02:41:27.824797 1806 heartbeater.cc:503] Master 127.2.251.62:45115 was elected leader, sending a full tablet report...
I20260811 02:41:27.825269 2054 raft_consensus.cc:348] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:27.825433 2054 raft_consensus.cc:374] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:27.825471 2054 raft_consensus.cc:729] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8920722b82c4d0a935c12b227a4de03, State: Initialized, Role: FOLLOWER
I20260811 02:41:27.825587 2054 consensus_queue.cc:260] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:27.825879 2054 ts_tablet_manager.cc:1428] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 02:41:27.825888 1587 heartbeater.cc:503] Master 127.2.251.62:45115 was elected leader, sending a full tablet report...
W20260811 02:41:27.859619 1808 tablet.cc:2369] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:41:27.915779 1589 tablet.cc:2369] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:28.001350 2063 raft_consensus.cc:482] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:28.001500 2063 raft_consensus.cc:504] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:28.001938 2063 leader_election.cc:288] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8920722b82c4d0a935c12b227a4de03 (127.2.251.1:32797), 82fe9ed83f2f40ae89d2f46ad442c339 (127.2.251.3:43625)
I20260811 02:41:28.005092 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.002130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:32797 (local address 127.2.251.2:43011)
0811 02:41:28.002183 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.002196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.002252 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.002461 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.002462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.002465 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.002555 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.002557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.003502 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.003503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.004255 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.004257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.004845 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.004847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.004928 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.004932 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.004985 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.005031 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 02:41:28.005141 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.002193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:43011 (local address 127.2.251.1:32797)
0811 02:41:28.002242 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.002244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.002265 (+ 21us) server_negotiation.cc:407] Connection header received
0811 02:41:28.002362 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.002363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.002377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.002446 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.002673 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.003410 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.004356 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.004751 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.004805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.004815 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.005021 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.005068 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:41:28.005268 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.002148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:43625 (local address 127.2.251.2:49351)
0811 02:41:28.002376 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.002386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.002440 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.002716 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.002717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.002719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.002798 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.002800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.003440 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.003441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.004358 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.004362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.005019 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.005021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.005096 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.005101 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.005156 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.005193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":53,"thread_start_us":144,"threads_started":1}
I20260811 02:41:28.005395 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.002289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:49351 (local address 127.2.251.3:43625)
0811 02:41:28.002333 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.002335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.002580 (+ 245us) server_negotiation.cc:407] Connection header received
0811 02:41:28.002610 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.002611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.002625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.002676 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.002879 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.003359 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.004472 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.004915 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.004994 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.005003 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.005291 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.005333 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:41:28.005605 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c892b9684ba2a14929bc34653254" candidate_uuid: "3e63841c300041c9892fa92c84580ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8920722b82c4d0a935c12b227a4de03" is_pre_election: true
I20260811 02:41:28.005678 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c892b9684ba2a14929bc34653254" candidate_uuid: "3e63841c300041c9892fa92c84580ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" is_pre_election: true
I20260811 02:41:28.005806 1514 raft_consensus.cc:2454] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e63841c300041c9892fa92c84580ec7 in term 0.
I20260811 02:41:28.005820 1951 raft_consensus.cc:2454] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e63841c300041c9892fa92c84580ec7 in term 0.
I20260811 02:41:28.006049 1619 leader_election.cc:302] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e63841c300041c9892fa92c84580ec7, a8920722b82c4d0a935c12b227a4de03; no voters:
I20260811 02:41:28.006177 2063 raft_consensus.cc:2781] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:28.006256 2063 raft_consensus.cc:482] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:28.006280 2063 raft_consensus.cc:3037] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.006971 2063 raft_consensus.cc:504] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:28.007094 2063 leader_election.cc:288] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [CANDIDATE]: Term 1 election: Requested vote from peers a8920722b82c4d0a935c12b227a4de03 (127.2.251.1:32797), 82fe9ed83f2f40ae89d2f46ad442c339 (127.2.251.3:43625)
I20260811 02:41:28.007314 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c892b9684ba2a14929bc34653254" candidate_uuid: "3e63841c300041c9892fa92c84580ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82fe9ed83f2f40ae89d2f46ad442c339"
I20260811 02:41:28.007314 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c892b9684ba2a14929bc34653254" candidate_uuid: "3e63841c300041c9892fa92c84580ec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8920722b82c4d0a935c12b227a4de03"
I20260811 02:41:28.007402 1514 raft_consensus.cc:3037] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.007402 1951 raft_consensus.cc:3037] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.008276 1514 raft_consensus.cc:2454] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e63841c300041c9892fa92c84580ec7 in term 1.
I20260811 02:41:28.008276 1951 raft_consensus.cc:2454] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e63841c300041c9892fa92c84580ec7 in term 1.
I20260811 02:41:28.008456 1619 leader_election.cc:302] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e63841c300041c9892fa92c84580ec7, a8920722b82c4d0a935c12b227a4de03; no voters:
I20260811 02:41:28.008561 2063 raft_consensus.cc:2781] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:28.008735 2063 raft_consensus.cc:686] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 3e63841c300041c9892fa92c84580ec7, State: Running, Role: LEADER
I20260811 02:41:28.008846 2063 consensus_queue.cc:237] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } }
I20260811 02:41:28.009706 1275 catalog_manager.cc:5495] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e63841c300041c9892fa92c84580ec7 (127.2.251.2). New cstate: current_term: 1 leader_uuid: "3e63841c300041c9892fa92c84580ec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e63841c300041c9892fa92c84580ec7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 41073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 } health_report { overall_health: UNKNOWN } } }
W20260811 02:41:28.028364 2025 tablet.cc:2369] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:28.045612 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.042601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.2.251.2:41073)
0811 02:41:28.042672 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.042673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.042681 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:28.042721 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.042721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.042743 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.042832 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:41:28.043261 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.043731 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.044620 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.044832 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.044887 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.045172 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:41:28.045308 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:41:28.045411 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.045518 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.045526 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 02:41:28.047596 1514 raft_consensus.cc:1261] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 1 FOLLOWER]: Refusing update from remote peer 3e63841c300041c9892fa92c84580ec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:28.047592 1951 raft_consensus.cc:1261] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 1 FOLLOWER]: Refusing update from remote peer 3e63841c300041c9892fa92c84580ec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:28.047966 2063 consensus_queue.cc:1035] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82fe9ed83f2f40ae89d2f46ad442c339" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:28.048115 2066 consensus_queue.cc:1035] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8920722b82c4d0a935c12b227a4de03" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:28.050788 2069 mvcc.cc:205] Tried to move back new op lower bound from 7317160296639987712 to 7317160296484728832. Current Snapshot: MvccSnapshot[applied={T|T < 7317160296639987712}]
I20260811 02:41:28.051533 2070 mvcc.cc:205] Tried to move back new op lower bound from 7317160296639987712 to 7317160296484728832. Current Snapshot: MvccSnapshot[applied={T|T < 7317160296639987712}]
I20260811 02:41:28.051605 2071 mvcc.cc:205] Tried to move back new op lower bound from 7317160296639987712 to 7317160296484728832. Current Snapshot: MvccSnapshot[applied={T|T < 7317160296639987712}]
I20260811 02:41:28.053499 1276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:28.053594 1276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:28.054415 1276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5388382f377c467bbe9ca7d0436e3346: Sending DeleteTablet for 3 replicas of tablet 91e4c892b9684ba2a14929bc34653254
I20260811 02:41:28.054767 1481 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e4c892b9684ba2a14929bc34653254 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:28 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:52150
I20260811 02:41:28.054849 1920 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e4c892b9684ba2a14929bc34653254 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:28 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49742
I20260811 02:41:28.055085 1688 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e4c892b9684ba2a14929bc34653254 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:28 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49620
I20260811 02:41:28.055315 2083 tablet_replica.cc:330] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: stopping tablet replica
I20260811 02:41:28.055421 2083 raft_consensus.cc:2229] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:41:28.055437 2082 tablet_replica.cc:330] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: stopping tablet replica
I20260811 02:41:28.055522 2083 raft_consensus.cc:2258] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:28.055546 2082 raft_consensus.cc:2229] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:28.055655 2082 raft_consensus.cc:2258] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:28.055742 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 1387
I20260811 02:41:28.055840 2081 tablet_replica.cc:330] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03: stopping tablet replica
I20260811 02:41:28.055938 2081 raft_consensus.cc:2229] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:28.055982 2083 ts_tablet_manager.cc:1905] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:28.056032 2081 raft_consensus.cc:2258] T 91e4c892b9684ba2a14929bc34653254 P a8920722b82c4d0a935c12b227a4de03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:28.056994 2082 ts_tablet_manager.cc:1905] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:28.057745 2083 ts_tablet_manager.cc:1918] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:28.057823 2083 log.cc:1196] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/91e4c892b9684ba2a14929bc34653254
I20260811 02:41:28.058101 2083 ts_tablet_manager.cc:1939] T 91e4c892b9684ba2a14929bc34653254 P 3e63841c300041c9892fa92c84580ec7: Deleting consensus metadata
I20260811 02:41:28.058537 1246 catalog_manager.cc:4841] TS 3e63841c300041c9892fa92c84580ec7 (127.2.251.2:41073): tablet 91e4c892b9684ba2a14929bc34653254 (table test-table [id=963c949ebaec44d5ab1848f41c3c159c]) successfully deleted
I20260811 02:41:28.058992 2082 ts_tablet_manager.cc:1918] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:28.059079 2082 log.cc:1196] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/91e4c892b9684ba2a14929bc34653254
I20260811 02:41:28.059393 2082 ts_tablet_manager.cc:1939] T 91e4c892b9684ba2a14929bc34653254 P 82fe9ed83f2f40ae89d2f46ad442c339: Deleting consensus metadata
I20260811 02:41:28.059785 1252 catalog_manager.cc:4841] TS 82fe9ed83f2f40ae89d2f46ad442c339 (127.2.251.3:43625): tablet 91e4c892b9684ba2a14929bc34653254 (table test-table [id=963c949ebaec44d5ab1848f41c3c159c]) successfully deleted
W20260811 02:41:28.070354 1253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:32797 (error 108)
I20260811 02:41:28.070992 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.070836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:32797 (local address 127.0.0.1:52156)
0811 02:41:28.070907 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.070953 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:32797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 02:41:28.071007 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 1594
W20260811 02:41:28.077818 1253 catalog_manager.cc:4584] TS a8920722b82c4d0a935c12b227a4de03 (127.2.251.1:32797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91e4c892b9684ba2a14929bc34653254: Network error: Client connection negotiation failed: client connection to 127.2.251.1:32797: connect: Connection refused (error 111)
I20260811 02:41:28.082007 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 1815
I20260811 02:41:28.089535 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 1229
2026-08-11T02:41:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2326 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T02:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:28Z Disabled control of system clock
I20260811 02:41:28.127702 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:44285
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:28.218351 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:28.218647 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:28.218672 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:28.220010 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:28.220068 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:28.220079 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:28.220091 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:28.220103 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:28.221648 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:44285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:44285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:28.221971 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:28.222294 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:28.229158 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:28.229139 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:28.229123 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:28.229534 2091 server_base.cc:1034] running on GCE node
I20260811 02:41:28.229856 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:28.230175 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:28.231349 2091 hybrid_clock.cc:648] HybridClock initialized: now 1786416088231341 us; error 39 us; skew 500 ppm
I20260811 02:41:28.232766 2091 webserver.cc:460] Webserver started at http://127.2.251.62:37951/ using document root <none> and password file <none>
I20260811 02:41:28.233028 2091 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:28.233083 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:28.233220 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:28.234390 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "025498be296241bfa3bc73fc428aed6c"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "ba68788bc62f9b7b3120c259a806e84c"
server_key_iv: "bee1a7bd760ab1aeeb63e2b366d31809"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.234741 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "025498be296241bfa3bc73fc428aed6c"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "ba68788bc62f9b7b3120c259a806e84c"
server_key_iv: "bee1a7bd760ab1aeeb63e2b366d31809"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.236152 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:28.237021 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.237254 2091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:28.237352 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "025498be296241bfa3bc73fc428aed6c"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "ba68788bc62f9b7b3120c259a806e84c"
server_key_iv: "bee1a7bd760ab1aeeb63e2b366d31809"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.237425 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:28.260421 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:28.260800 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:28.260946 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:28.264995 2091 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:44285
I20260811 02:41:28.265098 2186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:44285 every 8 connection(s)
I20260811 02:41:28.265409 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:28.266469 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:28.268855 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c: Bootstrap starting.
I20260811 02:41:28.269435 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:28.269678 2187 log.cc:824] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:28.270370 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c: No bootstrap required, opened a new log
I20260811 02:41:28.271926 2187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025498be296241bfa3bc73fc428aed6c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44285 } }
I20260811 02:41:28.272068 2187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:28.272090 2187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025498be296241bfa3bc73fc428aed6c, State: Initialized, Role: FOLLOWER
I20260811 02:41:28.272202 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025498be296241bfa3bc73fc428aed6c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44285 } }
I20260811 02:41:28.272276 2187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:28.272300 2187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:28.272331 2187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.272820 2187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025498be296241bfa3bc73fc428aed6c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44285 } }
I20260811 02:41:28.272909 2187 leader_election.cc:302] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025498be296241bfa3bc73fc428aed6c; no voters:
I20260811 02:41:28.273034 2187 leader_election.cc:288] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:28.273102 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:28.273275 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [term 1 LEADER]: Becoming Leader. State: Replica: 025498be296241bfa3bc73fc428aed6c, State: Running, Role: LEADER
I20260811 02:41:28.273273 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:28.273340 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 2091
I20260811 02:41:28.273382 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025498be296241bfa3bc73fc428aed6c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44285 } }
I20260811 02:41:28.273447 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:28.273633 3052 external_mini_cluster.cc:1428] Setting key 904252a1ec05b1511b0ae873822cc266
I20260811 02:41:28.273864 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025498be296241bfa3bc73fc428aed6c. Latest consensus state: current_term: 1 leader_uuid: "025498be296241bfa3bc73fc428aed6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025498be296241bfa3bc73fc428aed6c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44285 } } }
I20260811 02:41:28.273957 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:28.273936 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "025498be296241bfa3bc73fc428aed6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025498be296241bfa3bc73fc428aed6c" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 44285 } } }
I20260811 02:41:28.274022 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:28.274925 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:28.275440 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:28.277007 2203 catalog_manager.cc:1347] Generated new cluster ID: 5a806e583ac748acbdb017c517f0e1aa
I20260811 02:41:28.277064 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:28.279399 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.274324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.2.251.62:44285)
0811 02:41:28.274592 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.274599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.274625 (+ 26us) server_negotiation.cc:407] Connection header received
0811 02:41:28.274733 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.274738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.274812 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.274961 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:28.275306 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.276640 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.277488 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.277986 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.278095 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.278735 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:28.278739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:28.278739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:28.278913 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:28.278946 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:28.278957 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:28.279013 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.279081 (+ 68us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.279160 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":148,"threads_started":1}
I20260811 02:41:28.337653 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:28.338263 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:28.341197 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c: Generated new TSK 0
I20260811 02:41:28.341568 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:28.348021 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44285
--builtin_ntp_servers=127.2.251.20:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:28.444358 2229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:28.444612 2229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:28.444650 2229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:28.445899 2229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:28.445950 2229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:28.445992 2229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:28.447468 2229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:28.447753 2229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:28.448083 2229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:28.453338 2239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:28.453311 2237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:28.453347 2229 server_base.cc:1034] running on GCE node
W20260811 02:41:28.453298 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:28.453756 2229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:28.454011 2229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:28.455168 2229 hybrid_clock.cc:648] HybridClock initialized: now 1786416088455157 us; error 33 us; skew 500 ppm
I20260811 02:41:28.456549 2229 webserver.cc:460] Webserver started at http://127.2.251.1:38799/ using document root <none> and password file <none>
I20260811 02:41:28.456799 2229 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:28.456847 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:28.456964 2229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:28.458166 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "ac390f2eeaa44dfebcac0a020b23145a"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "d7aa6b0ffdeb818d7a77a9147565b37e"
server_key_iv: "df142f4f9fea2effc25835ab65887679"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.458524 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "ac390f2eeaa44dfebcac0a020b23145a"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "d7aa6b0ffdeb818d7a77a9147565b37e"
server_key_iv: "df142f4f9fea2effc25835ab65887679"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.459790 2229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:28.460595 2248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.460773 2229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:28.460856 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "ac390f2eeaa44dfebcac0a020b23145a"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "d7aa6b0ffdeb818d7a77a9147565b37e"
server_key_iv: "df142f4f9fea2effc25835ab65887679"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.460919 2229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:28.479856 2229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:28.480230 2229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:28.480496 2229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:28.480808 2229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:28.481169 2229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:28.481205 2229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.481238 2229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:28.481256 2229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.490736 2229 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:46109
I20260811 02:41:28.490784 2431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:46109 every 8 connection(s)
I20260811 02:41:28.491138 2229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:41:28.496592 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.491916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:46547 (local address 127.2.251.62:44285)
0811 02:41:28.492013 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.492015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.492219 (+ 204us) server_negotiation.cc:407] Connection header received
0811 02:41:28.492785 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.492786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.492801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.492866 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:28.493581 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.494164 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.495197 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.495415 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.495481 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.496356 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:28.496357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:28.496358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:28.496362 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:28.496378 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:28.496382 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:28.496430 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.496496 (+ 66us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.496527 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 02:41:28.496762 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.491867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44285 (local address 127.2.251.1:46547)
0811 02:41:28.492070 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.492111 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.492671 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.492947 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.492952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.492960 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:28.493473 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.493477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.494275 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.494276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.495090 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.495092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.495527 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.495528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.495669 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.496200 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:28.496200 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:28.496230 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:28.496257 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:28.496436 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:28.496439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:28.496442 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.496492 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.496579 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260811 02:41:28.496977 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 2229
I20260811 02:41:28.497076 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:41:28.497282 3052 external_mini_cluster.cc:1428] Setting key fd804125d7c1aba7505d833e5f4f9954
I20260811 02:41:28.497474 2432 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44285
I20260811 02:41:28.497575 2432 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:28.497812 2432 heartbeater.cc:511] Master 127.2.251.62:44285 requested a full tablet report, sending...
I20260811 02:41:28.498296 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44285
--builtin_ntp_servers=127.2.251.20:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 02:41:28.498319 2139 ts_manager.cc:194] Registered new tserver with Master: ac390f2eeaa44dfebcac0a020b23145a (127.2.251.1:46109)
I20260811 02:41:28.498986 2139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.1:46547
W20260811 02:41:28.589478 2437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:28.589768 2437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:28.589810 2437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:28.591035 2437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:28.591080 2437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:28.591117 2437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:28.592535 2437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:28.592815 2437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:28.593142 2437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:28.598486 2443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:28.598495 2444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:28.598513 2437 server_base.cc:1034] running on GCE node
W20260811 02:41:28.598493 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:28.598981 2437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:28.599323 2437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:28.600477 2437 hybrid_clock.cc:648] HybridClock initialized: now 1786416088600444 us; error 51 us; skew 500 ppm
I20260811 02:41:28.602051 2437 webserver.cc:460] Webserver started at http://127.2.251.2:42853/ using document root <none> and password file <none>
I20260811 02:41:28.602329 2437 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:28.602378 2437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:28.602504 2437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:28.603613 2437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "af7bbb06f63141f194fc712fd241f554"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "f1e08b55b23ba3d65c5e79a5b26454a6"
server_key_iv: "b1d6ff04705841c5e4af84477a021401"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.603962 2437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "af7bbb06f63141f194fc712fd241f554"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "f1e08b55b23ba3d65c5e79a5b26454a6"
server_key_iv: "b1d6ff04705841c5e4af84477a021401"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.605405 2437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:28.606297 2454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.606487 2437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:41:28.606585 2437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "af7bbb06f63141f194fc712fd241f554"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "f1e08b55b23ba3d65c5e79a5b26454a6"
server_key_iv: "b1d6ff04705841c5e4af84477a021401"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.606654 2437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:28.621070 2437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:28.621440 2437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:28.621562 2437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:28.621857 2437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:28.622213 2437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:28.622251 2437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.622274 2437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:28.622285 2437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.630396 2437 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:46117
I20260811 02:41:28.630512 2618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:46117 every 8 connection(s)
I20260811 02:41:28.630905 2437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:28.635886 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 2437
I20260811 02:41:28.636008 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:41:28.636212 3052 external_mini_cluster.cc:1428] Setting key dbcaa17f981189fc7674538f984e7e8c
I20260811 02:41:28.636884 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.631813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:57309 (local address 127.2.251.62:44285)
0811 02:41:28.631940 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.631943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.632136 (+ 193us) server_negotiation.cc:407] Connection header received
0811 02:41:28.632710 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.632711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.632724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.632781 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:28.633558 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.634136 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.635191 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.635388 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.635468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.636530 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:28.636532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:28.636532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:28.636538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:28.636556 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:28.636562 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:28.636609 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.636781 (+ 172us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.636814 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 02:41:28.637225 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.631776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44285 (local address 127.2.251.2:57309)
0811 02:41:28.632000 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.632045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.632599 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.632871 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.632876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.632886 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:28.633457 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.633462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.634244 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.634245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.635093 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.635096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.635487 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.635488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.635638 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.636353 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:28.636353 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:28.636397 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:28.636432 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:28.636649 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:28.636659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:28.636664 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.636749 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.636879 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260811 02:41:28.637461 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44285
--builtin_ntp_servers=127.2.251.20:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 02:41:28.637975 2620 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44285
I20260811 02:41:28.638090 2620 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:28.638303 2620 heartbeater.cc:511] Master 127.2.251.62:44285 requested a full tablet report, sending...
I20260811 02:41:28.638933 2139 ts_manager.cc:194] Registered new tserver with Master: af7bbb06f63141f194fc712fd241f554 (127.2.251.2:46117)
I20260811 02:41:28.639412 2139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.2:57309
W20260811 02:41:28.733919 2627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:28.734153 2627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:28.734175 2627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:28.735460 2627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:28.735515 2627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:28.735543 2627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:28.736999 2627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:44285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:28.737272 2627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:28.737625 2627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:28.742923 2636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:28.742928 2634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:28.742964 2633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:28.743098 2627 server_base.cc:1034] running on GCE node
I20260811 02:41:28.743386 2627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:28.743640 2627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:28.744841 2627 hybrid_clock.cc:648] HybridClock initialized: now 1786416088744829 us; error 31 us; skew 500 ppm
I20260811 02:41:28.746433 2627 webserver.cc:460] Webserver started at http://127.2.251.3:37989/ using document root <none> and password file <none>
I20260811 02:41:28.746834 2627 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:28.746901 2627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:28.747040 2627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:28.748392 2627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "6abf19367c404d1ca03751dbe451a9a1"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "c44de52bb499b2ed9ed1aee947ef7340"
server_key_iv: "805910af9a0e0afab61e8985560fbd92"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.748769 2627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "6abf19367c404d1ca03751dbe451a9a1"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "c44de52bb499b2ed9ed1aee947ef7340"
server_key_iv: "805910af9a0e0afab61e8985560fbd92"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.750259 2627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260811 02:41:28.751184 2645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.751408 2627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:28.751518 2627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "6abf19367c404d1ca03751dbe451a9a1"
format_stamp: "Formatted at 2026-08-11 02:41:28 on dist-test-slave-cvwc"
server_key: "c44de52bb499b2ed9ed1aee947ef7340"
server_key_iv: "805910af9a0e0afab61e8985560fbd92"
server_key_version: "encryptionkey@0"
I20260811 02:41:28.751587 2627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:28.774850 2627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:28.775213 2627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:28.775362 2627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:28.775609 2627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:28.775967 2627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:28.776003 2627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.776033 2627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:28.776050 2627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:28.782326 2627 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:43223
I20260811 02:41:28.782373 2828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:43223 every 8 connection(s)
I20260811 02:41:28.782713 2627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:28.785295 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 2627
I20260811 02:41:28.785396 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:41:28.785558 3052 external_mini_cluster.cc:1428] Setting key ee67cf019eb398c7b4fb84c36dc5596a
I20260811 02:41:28.788597 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.783478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:36077 (local address 127.2.251.62:44285)
0811 02:41:28.783537 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.783539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.783754 (+ 215us) server_negotiation.cc:407] Connection header received
0811 02:41:28.784300 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.784302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.784314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.784369 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:28.785124 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.785894 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.787089 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.787291 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.787354 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.788300 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:28.788301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:28.788302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:28.788307 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:28.788324 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:28.788328 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:28.788381 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.788504 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.788534 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 02:41:28.788810 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.783407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:44285 (local address 127.2.251.3:36077)
0811 02:41:28.783632 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.783668 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.784184 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.784433 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.784436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.784445 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:28.784968 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.784972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.785992 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.785993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.787001 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.787005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.787397 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.787398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.787534 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.788096 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:28.788096 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:28.788126 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:28.788152 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:28.788415 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:28.788422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:28.788425 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.788501 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.788591 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260811 02:41:28.789218 2829 heartbeater.cc:347] Connected to a master server at 127.2.251.62:44285
I20260811 02:41:28.789315 2829 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:28.789485 2829 heartbeater.cc:511] Master 127.2.251.62:44285 requested a full tablet report, sending...
I20260811 02:41:28.789857 2139 ts_manager.cc:194] Registered new tserver with Master: 6abf19367c404d1ca03751dbe451a9a1 (127.2.251.3:43223)
I20260811 02:41:28.790192 2139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.3:36077
I20260811 02:41:28.797904 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:41:28.802515 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.799323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.2.251.62:44285)
0811 02:41:28.799388 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.799391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.799483 (+ 92us) server_negotiation.cc:407] Connection header received
0811 02:41:28.799549 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.799550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.799563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.799611 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:28.799748 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.800261 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.801074 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.801344 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.801438 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.802228 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:28.802230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:28.802230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:28.802235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:28.802248 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:28.802252 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:28.802327 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.802413 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.802427 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:41:28.805058 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:41:28.807026 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:41:28.815042 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.2.251.1:46109)
0811 02:41:28.810706 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.810712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.811217 (+ 505us) server_negotiation.cc:407] Connection header received
0811 02:41:28.811287 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.811290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.811322 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.811417 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.811796 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.812546 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.813834 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.814608 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814703 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.814818 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.814898 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.814946 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":144,"threads_started":1}
I20260811 02:41:28.815032 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.810124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:46109 (local address 127.0.0.1:42106)
0811 02:41:28.811130 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.811139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.811195 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.811617 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.811619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.811623 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.811705 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.811709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.812642 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.812643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.813702 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.813705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.814711 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.814788 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.814805 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.814879 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.814937 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":6,"thread_start_us":100,"threads_started":1}
I20260811 02:41:28.815150 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.810094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:43223 (local address 127.0.0.1:53750)
0811 02:41:28.810256 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.810302 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.810383 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.811070 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.811074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.811077 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.811193 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.811196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.812507 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.812509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.813919 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.813923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.814835 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.814924 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.814929 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.814963 (+ 34us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.815007 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":27,"thread_start_us":93,"threads_started":1}
I20260811 02:41:28.815308 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.810124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:46117 (local address 127.0.0.1:38566)
0811 02:41:28.810956 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.810970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.811048 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.811659 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.811660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.811663 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.811723 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.811726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.812955 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.812956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.814128 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.814130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.814907 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.814958 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.814962 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.815005 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.815040 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":7,"thread_start_us":101,"threads_started":1}
I20260811 02:41:28.815315 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.810692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.2.251.2:46117)
0811 02:41:28.810950 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.810954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.811373 (+ 419us) server_negotiation.cc:407] Connection header received
0811 02:41:28.811425 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.811427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.811442 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.811674 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.811839 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.812861 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814205 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.814696 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814766 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.814869 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.815080 (+ 211us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.815121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":145,"threads_started":1}
I20260811 02:41:28.815769 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.2.251.3:43223)
0811 02:41:28.810794 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.810835 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.810854 (+ 19us) server_negotiation.cc:407] Connection header received
0811 02:41:28.810913 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.810916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.810937 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.811059 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.811297 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.812396 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.814020 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.814725 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.815083 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.815582 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.815613 (+ 31us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.815661 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260811 02:41:28.816447 2525 tablet_service.cc:1467] Processing CreateTablet for tablet ca8b554fc6884ffa8b123791cbd6e245 (DEFAULT_TABLE table=test-table [id=bbf3e2fa28ee419d9393cb6a8e80c188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:28.816450 2717 tablet_service.cc:1467] Processing CreateTablet for tablet ca8b554fc6884ffa8b123791cbd6e245 (DEFAULT_TABLE table=test-table [id=bbf3e2fa28ee419d9393cb6a8e80c188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:28.816862 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8b554fc6884ffa8b123791cbd6e245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:28.816862 2717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8b554fc6884ffa8b123791cbd6e245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:28.816890 2320 tablet_service.cc:1467] Processing CreateTablet for tablet ca8b554fc6884ffa8b123791cbd6e245 (DEFAULT_TABLE table=test-table [id=bbf3e2fa28ee419d9393cb6a8e80c188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:28.817253 2320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8b554fc6884ffa8b123791cbd6e245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:28.820685 2855 tablet_bootstrap.cc:492] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Bootstrap starting.
I20260811 02:41:28.820998 2856 tablet_bootstrap.cc:492] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Bootstrap starting.
I20260811 02:41:28.821383 2855 tablet_bootstrap.cc:654] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:28.821532 2856 tablet_bootstrap.cc:654] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:28.821825 2856 log.cc:824] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:28.822057 2855 log.cc:824] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:28.822156 2857 tablet_bootstrap.cc:492] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Bootstrap starting.
I20260811 02:41:28.822795 2857 tablet_bootstrap.cc:654] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:28.823093 2857 log.cc:824] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:28.823601 2856 tablet_bootstrap.cc:492] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: No bootstrap required, opened a new log
I20260811 02:41:28.823671 2856 ts_tablet_manager.cc:1397] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 02:41:28.823628 2855 tablet_bootstrap.cc:492] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: No bootstrap required, opened a new log
I20260811 02:41:28.823827 2855 ts_tablet_manager.cc:1397] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:41:28.825609 2856 raft_consensus.cc:348] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.825814 2856 raft_consensus.cc:374] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:28.825870 2856 raft_consensus.cc:729] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7bbb06f63141f194fc712fd241f554, State: Initialized, Role: FOLLOWER
I20260811 02:41:28.826020 2856 consensus_queue.cc:260] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.826238 2855 raft_consensus.cc:348] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.826426 2856 ts_tablet_manager.cc:1428] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:41:28.826442 2855 raft_consensus.cc:374] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:28.826498 2855 raft_consensus.cc:729] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac390f2eeaa44dfebcac0a020b23145a, State: Initialized, Role: FOLLOWER
I20260811 02:41:28.826588 2620 heartbeater.cc:503] Master 127.2.251.62:44285 was elected leader, sending a full tablet report...
I20260811 02:41:28.826643 2855 consensus_queue.cc:260] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.826776 2857 tablet_bootstrap.cc:492] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: No bootstrap required, opened a new log
I20260811 02:41:28.826867 2857 ts_tablet_manager.cc:1397] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 02:41:28.826979 2855 ts_tablet_manager.cc:1428] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260811 02:41:28.827256 2432 heartbeater.cc:503] Master 127.2.251.62:44285 was elected leader, sending a full tablet report...
I20260811 02:41:28.831082 2857 raft_consensus.cc:348] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.831295 2857 raft_consensus.cc:374] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:28.831318 2857 raft_consensus.cc:729] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6abf19367c404d1ca03751dbe451a9a1, State: Initialized, Role: FOLLOWER
I20260811 02:41:28.831444 2857 consensus_queue.cc:260] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.831792 2857 ts_tablet_manager.cc:1428] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 02:41:28.831935 2829 heartbeater.cc:503] Master 127.2.251.62:44285 was elected leader, sending a full tablet report...
W20260811 02:41:28.881590 2625 tablet.cc:2369] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:28.961473 2864 raft_consensus.cc:482] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:28.961615 2864 raft_consensus.cc:504] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.962059 2864 leader_election.cc:288] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af7bbb06f63141f194fc712fd241f554 (127.2.251.2:46117), ac390f2eeaa44dfebcac0a020b23145a (127.2.251.1:46109)
I20260811 02:41:28.962673 2862 raft_consensus.cc:482] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:28.962872 2862 raft_consensus.cc:504] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.964504 2862 leader_election.cc:288] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6abf19367c404d1ca03751dbe451a9a1 (127.2.251.3:43223), ac390f2eeaa44dfebcac0a020b23145a (127.2.251.1:46109)
I20260811 02:41:28.966090 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.962457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:58119 (local address 127.2.251.2:46117)
0811 02:41:28.962510 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.962512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.962605 (+ 93us) server_negotiation.cc:407] Connection header received
0811 02:41:28.962663 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.962664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.962676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.962723 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.962946 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.963798 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.965123 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.965667 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.965740 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.965752 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.965979 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.966019 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:41:28.966351 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.962332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:46117 (local address 127.2.251.3:58119)
0811 02:41:28.962558 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.962566 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.962612 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.962802 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.962803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.962806 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.962874 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.962876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.963923 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.963924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.965026 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.965030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.965823 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.965824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.965900 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.965904 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.966225 (+ 321us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.966269 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":34,"thread_start_us":116,"threads_started":1}
I20260811 02:41:28.966925 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.962533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:57351 (local address 127.2.251.1:46109)
0811 02:41:28.962574 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.962576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.962584 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:28.962617 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.962618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.962630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.962700 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.963004 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.964278 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.965867 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.966390 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.966461 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.966472 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.966803 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.966847 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:41:28.966954 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.962332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:46109 (local address 127.2.251.3:57351)
0811 02:41:28.962402 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.962414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.962471 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.962716 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.962717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.962718 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.962893 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.962897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.964530 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.964531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.965761 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.965765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.966506 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.966508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.966585 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.966588 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.966777 (+ 189us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.966861 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 02:41:28.967562 2357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "6abf19367c404d1ca03751dbe451a9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac390f2eeaa44dfebcac0a020b23145a" is_pre_election: true
I20260811 02:41:28.967829 2357 raft_consensus.cc:2454] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6abf19367c404d1ca03751dbe451a9a1 in term 0.
I20260811 02:41:28.967841 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "6abf19367c404d1ca03751dbe451a9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7bbb06f63141f194fc712fd241f554" is_pre_election: true
I20260811 02:41:28.967960 2552 raft_consensus.cc:2454] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6abf19367c404d1ca03751dbe451a9a1 in term 0.
I20260811 02:41:28.968119 2652 leader_election.cc:302] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6abf19367c404d1ca03751dbe451a9a1, ac390f2eeaa44dfebcac0a020b23145a; no voters:
I20260811 02:41:28.968327 2864 raft_consensus.cc:2781] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:28.968385 2864 raft_consensus.cc:482] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:28.968408 2864 raft_consensus.cc:3037] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.968650 2867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.964504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:50851 (local address 127.2.251.1:46109)
0811 02:41:28.964775 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.964778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.964789 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:41:28.964815 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.964816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.964827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.964905 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.965120 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.965868 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.967605 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.968289 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.968351 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.968362 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.968543 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.968581 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":80,"threads_started":1}
I20260811 02:41:28.968643 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.963630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:43223 (local address 127.2.251.2:43649)
0811 02:41:28.963867 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.963878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.964254 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.965114 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.965115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.965117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.965180 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.965182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.965997 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.965998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.967332 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.967337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.968344 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.968345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.968433 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.968437 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.968487 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.968541 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":3}
I20260811 02:41:28.968717 2866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.963693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:46109 (local address 127.2.251.2:50851)
0811 02:41:28.963886 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:28.963896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:28.963941 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:28.964951 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:28.964952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:28.964955 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:28.965032 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.965035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.965950 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.965950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.967521 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:28.967524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.968395 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.968396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:28.968478 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.968482 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:28.968537 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 02:41:28.968578 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":40,"thread_start_us":114,"threads_started":1}
I20260811 02:41:28.968940 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.964457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:43649 (local address 127.2.251.3:43223)
0811 02:41:28.964981 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.964983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.964993 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:41:28.965016 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.965017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.965025 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.965234 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:28.965260 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.965915 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.967759 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:28.968257 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:28.968325 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:28.968335 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:28.968832 (+ 497us) server_negotiation.cc:299] Negotiation successful
0811 02:41:28.968879 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
I20260811 02:41:28.969208 2750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "af7bbb06f63141f194fc712fd241f554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abf19367c404d1ca03751dbe451a9a1" is_pre_election: true
I20260811 02:41:28.969236 2357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "af7bbb06f63141f194fc712fd241f554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac390f2eeaa44dfebcac0a020b23145a" is_pre_election: true
I20260811 02:41:28.969316 2357 raft_consensus.cc:2454] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7bbb06f63141f194fc712fd241f554 in term 0.
I20260811 02:41:28.969476 2459 leader_election.cc:302] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac390f2eeaa44dfebcac0a020b23145a, af7bbb06f63141f194fc712fd241f554; no voters:
I20260811 02:41:28.969517 2864 raft_consensus.cc:504] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.969620 2862 raft_consensus.cc:2781] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:28.969666 2862 raft_consensus.cc:482] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:28.969672 2750 raft_consensus.cc:2379] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af7bbb06f63141f194fc712fd241f554 in current term 1: Already voted for candidate 6abf19367c404d1ca03751dbe451a9a1 in this term.
I20260811 02:41:28.969688 2862 raft_consensus.cc:3037] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.970031 2357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "6abf19367c404d1ca03751dbe451a9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac390f2eeaa44dfebcac0a020b23145a"
I20260811 02:41:28.970124 2357 raft_consensus.cc:3037] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:28.970455 2862 raft_consensus.cc:504] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.970574 2862 leader_election.cc:288] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [CANDIDATE]: Term 1 election: Requested vote from peers 6abf19367c404d1ca03751dbe451a9a1 (127.2.251.3:43223), ac390f2eeaa44dfebcac0a020b23145a (127.2.251.1:46109)
I20260811 02:41:28.970709 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "6abf19367c404d1ca03751dbe451a9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7bbb06f63141f194fc712fd241f554"
I20260811 02:41:28.970789 2552 raft_consensus.cc:2379] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6abf19367c404d1ca03751dbe451a9a1 in current term 1: Already voted for candidate af7bbb06f63141f194fc712fd241f554 in this term.
I20260811 02:41:28.970785 2356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "af7bbb06f63141f194fc712fd241f554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac390f2eeaa44dfebcac0a020b23145a"
I20260811 02:41:28.970922 2357 raft_consensus.cc:2454] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6abf19367c404d1ca03751dbe451a9a1 in term 1.
I20260811 02:41:28.971046 2864 leader_election.cc:288] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [CANDIDATE]: Term 1 election: Requested vote from peers af7bbb06f63141f194fc712fd241f554 (127.2.251.2:46117), ac390f2eeaa44dfebcac0a020b23145a (127.2.251.1:46109)
I20260811 02:41:28.971100 2652 leader_election.cc:302] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6abf19367c404d1ca03751dbe451a9a1, ac390f2eeaa44dfebcac0a020b23145a; no voters: af7bbb06f63141f194fc712fd241f554
I20260811 02:41:28.971109 2750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8b554fc6884ffa8b123791cbd6e245" candidate_uuid: "af7bbb06f63141f194fc712fd241f554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6abf19367c404d1ca03751dbe451a9a1"
I20260811 02:41:28.971203 2750 raft_consensus.cc:2379] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af7bbb06f63141f194fc712fd241f554 in current term 1: Already voted for candidate 6abf19367c404d1ca03751dbe451a9a1 in this term.
I20260811 02:41:28.971237 2864 raft_consensus.cc:2781] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:28.971338 2457 leader_election.cc:302] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af7bbb06f63141f194fc712fd241f554; no voters: 6abf19367c404d1ca03751dbe451a9a1, ac390f2eeaa44dfebcac0a020b23145a
I20260811 02:41:28.971414 2864 raft_consensus.cc:686] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 LEADER]: Becoming Leader. State: Replica: 6abf19367c404d1ca03751dbe451a9a1, State: Running, Role: LEADER
I20260811 02:41:28.971436 2862 raft_consensus.cc:2726] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260811 02:41:28.971511 2864 consensus_queue.cc:237] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } }
I20260811 02:41:28.972445 2139 catalog_manager.cc:5495] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6abf19367c404d1ca03751dbe451a9a1 (127.2.251.3). New cstate: current_term: 1 leader_uuid: "6abf19367c404d1ca03751dbe451a9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6abf19367c404d1ca03751dbe451a9a1" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 } health_report { overall_health: UNKNOWN } } }
W20260811 02:41:28.992280 2433 tablet.cc:2369] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:29.002488 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:28.999579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.2.251.3:43223)
0811 02:41:28.999628 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:28.999630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:28.999639 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:41:28.999709 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:28.999709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:28.999729 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:28.999791 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:41:28.999917 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:29.000388 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:29.001453 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:29.001667 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:29.001816 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:29.002058 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:41:29.002163 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:41:29.002250 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:29.002353 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 02:41:29.002364 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:41:29.005656 2357 raft_consensus.cc:1261] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 1 FOLLOWER]: Refusing update from remote peer 6abf19367c404d1ca03751dbe451a9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:29.005656 2552 raft_consensus.cc:1261] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 1 FOLLOWER]: Refusing update from remote peer 6abf19367c404d1ca03751dbe451a9a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:29.006034 2864 consensus_queue.cc:1035] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac390f2eeaa44dfebcac0a020b23145a" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:29.006193 2868 consensus_queue.cc:1035] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7bbb06f63141f194fc712fd241f554" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:29.008801 2873 mvcc.cc:205] Tried to move back new op lower bound from 7317160300560879616 to 7317160300427866112. Current Snapshot: MvccSnapshot[applied={T|T < 7317160300560879616}]
I20260811 02:41:29.009865 2875 mvcc.cc:205] Tried to move back new op lower bound from 7317160300560879616 to 7317160300427866112. Current Snapshot: MvccSnapshot[applied={T|T < 7317160300560879616}]
I20260811 02:41:29.009971 2874 mvcc.cc:205] Tried to move back new op lower bound from 7317160300560879616 to 7317160300427866112. Current Snapshot: MvccSnapshot[applied={T|T < 7317160300560879616 or (T in {7317160300560879616})}]
I20260811 02:41:29.012478 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:29.012583 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:29.013697 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 025498be296241bfa3bc73fc428aed6c: Sending DeleteTablet for 3 replicas of tablet ca8b554fc6884ffa8b123791cbd6e245
I20260811 02:41:29.014180 2717 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8b554fc6884ffa8b123791cbd6e245 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:29 UTC) from {username='slave'} at 127.0.0.1:53750
I20260811 02:41:29.014281 2320 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8b554fc6884ffa8b123791cbd6e245 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:29 UTC) from {username='slave'} at 127.0.0.1:42106
I20260811 02:41:29.014628 2888 tablet_replica.cc:330] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: stopping tablet replica
I20260811 02:41:29.014822 2888 raft_consensus.cc:2229] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:29.014964 2888 raft_consensus.cc:2258] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:29.015375 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8b554fc6884ffa8b123791cbd6e245 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:29 UTC) from {username='slave'} at 127.0.0.1:38566
I20260811 02:41:29.015461 2888 ts_tablet_manager.cc:1905] T ca8b554fc6884ffa8b123791cbd6e245 P ac390f2eeaa44dfebcac0a020b23145a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:29.015652 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 2229
I20260811 02:41:29.015667 2889 tablet_replica.cc:330] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: stopping tablet replica
I20260811 02:41:29.015772 2889 raft_consensus.cc:2229] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:29.015867 2889 raft_consensus.cc:2258] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:29.040307 2887 tablet_replica.cc:330] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: stopping tablet replica
I20260811 02:41:29.040504 2887 raft_consensus.cc:2229] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:41:29.040648 2887 raft_consensus.cc:2258] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:29.041126 2887 ts_tablet_manager.cc:1905] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:29.041167 2889 ts_tablet_manager.cc:1905] T ca8b554fc6884ffa8b123791cbd6e245 P af7bbb06f63141f194fc712fd241f554: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 02:41:29.042516 2118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:46109 (error 108)
I20260811 02:41:29.043169 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 2437
I20260811 02:41:29.048913 2887 ts_tablet_manager.cc:1918] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:29.049106 2887 log.cc:1196] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/ca8b554fc6884ffa8b123791cbd6e245
I20260811 02:41:29.049500 2887 ts_tablet_manager.cc:1939] T ca8b554fc6884ffa8b123791cbd6e245 P 6abf19367c404d1ca03751dbe451a9a1: Deleting consensus metadata
I20260811 02:41:29.095203 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:29.095000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:46109 (local address 127.0.0.1:42120)
0811 02:41:29.095075 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:29.095145 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 02:41:29.095268 2116 catalog_manager.cc:4841] TS 6abf19367c404d1ca03751dbe451a9a1 (127.2.251.3:43223): tablet ca8b554fc6884ffa8b123791cbd6e245 (table test-table [id=bbf3e2fa28ee419d9393cb6a8e80c188]) successfully deleted
W20260811 02:41:29.097126 2118 catalog_manager.cc:4584] TS ac390f2eeaa44dfebcac0a020b23145a (127.2.251.1:46109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca8b554fc6884ffa8b123791cbd6e245: Network error: Client connection negotiation failed: client connection to 127.2.251.1:46109: connect: Connection refused (error 111)
I20260811 02:41:29.103803 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 2627
I20260811 02:41:29.103869 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:29.103734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:46117 (local address 127.0.0.1:38580)
0811 02:41:29.103798 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:29.103831 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.2:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 02:41:29.115082 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 2091
2026-08-11T02:41:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1024 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:29 dist-test-slave-cvwc krb5kdc[2894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:29 dist-test-slave-cvwc krb5kdc[2894](info): set up 2 sockets
Aug 11 02:41:29 dist-test-slave-cvwc krb5kdc[2894](info): commencing operation
krb5kdc: starting...
W20260811 02:41:30.323324 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:30.457624 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38573
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:38573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:30.547309 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:30.547574 2908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:30.547600 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:30.548934 2908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:30.549013 2908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:30.549029 2908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:30.549043 2908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:30.549057 2908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:30.550864 2908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:38573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:38573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:30.551168 2908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:30.551486 2908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:30.556921 2917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:30.556918 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:30.556957 2908 server_base.cc:1034] running on GCE node
W20260811 02:41:30.556986 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:30.557446 2908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:30.557811 2908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:30.558981 2908 hybrid_clock.cc:648] HybridClock initialized: now 1786416090558942 us; error 59 us; skew 500 ppm
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:30.561827 2908 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:30.562301 2908 webserver.cc:460] Webserver started at http://127.2.251.62:36487/ using document root <none> and password file <none>
I20260811 02:41:30.562564 2908 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:30.562615 2908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:30.562731 2908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:30.563966 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "dbeaa4bd7630473780b209ac261e823a"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "49e0e2ad50b3ac8a53882c7d2f05d563"
server_key_iv: "8a121cde597adacab6ed7e9e77f0ae8c"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.564327 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "dbeaa4bd7630473780b209ac261e823a"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "49e0e2ad50b3ac8a53882c7d2f05d563"
server_key_iv: "8a121cde597adacab6ed7e9e77f0ae8c"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.565760 2908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:30.566629 2939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.566917 2908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:30.567026 2908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "dbeaa4bd7630473780b209ac261e823a"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "49e0e2ad50b3ac8a53882c7d2f05d563"
server_key_iv: "8a121cde597adacab6ed7e9e77f0ae8c"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.567107 2908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:30.582885 2908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:30.585202 2908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:30.585335 2908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:30.589607 2908 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:38573
I20260811 02:41:30.589660 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:38573 every 8 connection(s)
I20260811 02:41:30.590080 2908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:30.591048 3011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:30.593623 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a: Bootstrap starting.
I20260811 02:41:30.594209 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 2908
I20260811 02:41:30.594339 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:30.594329 3011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:30.594545 3052 external_mini_cluster.cc:1428] Setting key 63cac8877a9986a079a20657052fff49
I20260811 02:41:30.594630 3011 log.cc:824] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:30.595553 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a: No bootstrap required, opened a new log
I20260811 02:41:30.597956 3011 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeaa4bd7630473780b209ac261e823a" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38573 } }
I20260811 02:41:30.598150 3011 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:30.598191 3011 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbeaa4bd7630473780b209ac261e823a, State: Initialized, Role: FOLLOWER
I20260811 02:41:30.598326 3011 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeaa4bd7630473780b209ac261e823a" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38573 } }
I20260811 02:41:30.598419 3011 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:30.598456 3011 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:30.598518 3011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:30.599242 3011 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeaa4bd7630473780b209ac261e823a" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38573 } }
I20260811 02:41:30.599341 3011 leader_election.cc:302] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbeaa4bd7630473780b209ac261e823a; no voters:
I20260811 02:41:30.599542 3011 leader_election.cc:288] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:30.599581 3020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:30.599752 3020 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [term 1 LEADER]: Becoming Leader. State: Replica: dbeaa4bd7630473780b209ac261e823a, State: Running, Role: LEADER
I20260811 02:41:30.599907 3011 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:30.599946 3020 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeaa4bd7630473780b209ac261e823a" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38573 } }
I20260811 02:41:30.600607 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbeaa4bd7630473780b209ac261e823a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeaa4bd7630473780b209ac261e823a" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38573 } } }
I20260811 02:41:30.600793 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:30.600589 3021 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbeaa4bd7630473780b209ac261e823a. Latest consensus state: current_term: 1 leader_uuid: "dbeaa4bd7630473780b209ac261e823a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeaa4bd7630473780b209ac261e823a" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 38573 } } }
I20260811 02:41:30.601099 3021 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:30.601336 3034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:30.601887 3034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:30.604017 3034 catalog_manager.cc:1347] Generated new cluster ID: b932c74df032440bbf563e1b1b77da37
I20260811 02:41:30.604070 3034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:30.606168 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:30.595143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.2.251.62:38573)
0811 02:41:30.595326 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:30.595336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:30.595368 (+ 32us) server_negotiation.cc:407] Connection header received
0811 02:41:30.595937 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:30.595945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:30.596000 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:30.596123 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:30.596943 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.597914 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.598942 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.599424 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.599544 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:30.602567 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:30.602571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:30.602572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:30.602586 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:30.604444 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:30.604938 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:30.604941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:30.604942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:30.605040 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:30.605229 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:30.605231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:30.605231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:30.605493 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:30.605655 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:30.605794 (+ 139us) server_negotiation.cc:299] Negotiation successful
0811 02:41:30.605939 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260811 02:41:30.621632 3034 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:30.622572 3034 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:30.630376 3034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a: Generated new TSK 0
I20260811 02:41:30.630620 3034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.251.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
Entry for principal kudu/127.2.251.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab.
I20260811 02:41:30.662461 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38573
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:30.757040 3061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:30.757296 3061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:30.757324 3061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:30.758663 3061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:30.758725 3061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:30.758754 3061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:30.760458 3061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.1.keytab
--principal=kudu/127.2.251.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:30.760725 3061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:30.760960 3061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:30.766366 3069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:30.766387 3066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:30.766366 3072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:30.766551 3061 server_base.cc:1034] running on GCE node
I20260811 02:41:30.766836 3061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:30.767092 3061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:30.768252 3061 hybrid_clock.cc:648] HybridClock initialized: now 1786416090768237 us; error 30 us; skew 500 ppm
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:30.771376 3061 init.cc:377] Logged in from keytab as kudu/127.2.251.1@KRBTEST.COM (short username kudu)
I20260811 02:41:30.771867 3061 webserver.cc:460] Webserver started at http://127.2.251.1:37129/ using document root <none> and password file <none>
I20260811 02:41:30.772241 3061 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:30.772341 3061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:30.772581 3061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:30.773824 3061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "806f980071d84cd1bfd84965385436f5"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "f7d09405d5b6a9382a31f735e592e168"
server_key_iv: "1540aeddd916c74c7eca71ddcbc4b5cc"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.774173 3061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "806f980071d84cd1bfd84965385436f5"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "f7d09405d5b6a9382a31f735e592e168"
server_key_iv: "1540aeddd916c74c7eca71ddcbc4b5cc"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.775553 3061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:41:30.776482 3083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.776711 3061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:30.776818 3061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "806f980071d84cd1bfd84965385436f5"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "f7d09405d5b6a9382a31f735e592e168"
server_key_iv: "1540aeddd916c74c7eca71ddcbc4b5cc"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.776887 3061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:30.794390 3061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:30.797178 3061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:30.797369 3061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:30.797646 3061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:30.798041 3061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:30.798079 3061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.798110 3061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:30.798126 3061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.804452 3061 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:42527
I20260811 02:41:30.804615 3254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:42527 every 8 connection(s)
I20260811 02:41:30.804873 3061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:41:30.810616 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3061
I20260811 02:41:30.810762 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:41:30.811030 3052 external_mini_cluster.cc:1428] Setting key ddfabe2fff9c8312001bdd1fcfb8cb42
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.1@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:30.815653 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:30.805610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38573 (local address 127.2.251.1:60253)
0811 02:41:30.805788 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:30.805831 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:30.806350 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:30.806624 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:30.806628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:30.807228 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:30.807441 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:30.807446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.808156 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.808158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:30.808984 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:30.808986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.809345 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.809346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:30.809473 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:30.810065 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:30.810075 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:30.811809 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:30.813831 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:30.813840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:30.813843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:30.814207 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:30.814477 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:30.814478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:30.814478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:30.814703 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:30.815006 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:30.815012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:30.815200 (+ 188us) client_negotiation.cc:769] Sending connection context
0811 02:41:30.815325 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 02:41:30.815472 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260811 02:41:30.815690 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:30.805641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:60253 (local address 127.2.251.62:38573)
0811 02:41:30.805731 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:30.805733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:30.805911 (+ 178us) server_negotiation.cc:407] Connection header received
0811 02:41:30.806467 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:30.806469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:30.806482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:30.806537 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:30.807540 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.808065 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.809069 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.809268 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.809324 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:30.811935 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:30.811938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:30.811939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:30.811952 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:30.813703 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:30.814329 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:30.814330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:30.814330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:30.814408 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:30.814797 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:30.814798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:30.814798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:30.814935 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:30.815002 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:30.815441 (+ 439us) server_negotiation.cc:299] Negotiation successful
0811 02:41:30.815595 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:41:30.816391 3258 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38573
I20260811 02:41:30.816524 3258 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:30.816781 3258 heartbeater.cc:511] Master 127.2.251.62:38573 requested a full tablet report, sending...
I20260811 02:41:30.817458 2958 ts_manager.cc:194] Registered new tserver with Master: 806f980071d84cd1bfd84965385436f5 (127.2.251.1:42527)
I20260811 02:41:30.818215 2958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.1@KRBTEST.COM'} at 127.2.251.1:60253
WARNING: no policy specified for kudu/127.2.251.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
Entry for principal kudu/127.2.251.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab.
I20260811 02:41:30.841769 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38573
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:30.932235 3266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:30.932459 3266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:30.932484 3266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:30.933624 3266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:30.933663 3266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:30.933688 3266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:30.935250 3266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.2.keytab
--principal=kudu/127.2.251.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:30.935470 3266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:30.935675 3266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:41:30.941203 3266 server_base.cc:1034] running on GCE node
W20260811 02:41:30.941210 3273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:30.941382 3276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:30.941427 3272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:30.941687 3266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:30.941975 3266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:30.943130 3266 hybrid_clock.cc:648] HybridClock initialized: now 1786416090943115 us; error 29 us; skew 500 ppm
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:30.945834 3266 init.cc:377] Logged in from keytab as kudu/127.2.251.2@KRBTEST.COM (short username kudu)
I20260811 02:41:30.946322 3266 webserver.cc:460] Webserver started at http://127.2.251.2:34793/ using document root <none> and password file <none>
I20260811 02:41:30.946542 3266 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:30.946583 3266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:30.946676 3266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:30.947687 3266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "0b83d7d25d2a4f84bb7a86b87dac1495"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "3f5696b4083082e9c06b43cfee3c00ae"
server_key_iv: "dde6d73cbf2bfd12db2cbb084003a90b"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.948069 3266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "0b83d7d25d2a4f84bb7a86b87dac1495"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "3f5696b4083082e9c06b43cfee3c00ae"
server_key_iv: "dde6d73cbf2bfd12db2cbb084003a90b"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.949651 3266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:30.950562 3288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.950764 3266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:30.950853 3266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "0b83d7d25d2a4f84bb7a86b87dac1495"
format_stamp: "Formatted at 2026-08-11 02:41:30 on dist-test-slave-cvwc"
server_key: "3f5696b4083082e9c06b43cfee3c00ae"
server_key_iv: "dde6d73cbf2bfd12db2cbb084003a90b"
server_key_version: "encryptionkey@0"
I20260811 02:41:30.950924 3266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:30.962440 3266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:30.964944 3266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:30.965072 3266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:30.965281 3266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:30.965592 3266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:30.965626 3266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.965651 3266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:30.965673 3266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:30.973268 3266 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:44047
I20260811 02:41:30.973312 3455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:44047 every 8 connection(s)
I20260811 02:41:30.973668 3266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:30.979513 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3266
I20260811 02:41:30.979648 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:41:30.979944 3052 external_mini_cluster.cc:1428] Setting key 157cbc9e221aa8c3ea4169e5c4162a84
Aug 11 02:41:30 dist-test-slave-cvwc krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.2@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:30.985714 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:30.974514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:36297 (local address 127.2.251.62:38573)
0811 02:41:30.974598 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:30.974600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:30.974783 (+ 183us) server_negotiation.cc:407] Connection header received
0811 02:41:30.975362 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:30.975363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:30.975374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:30.975436 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:30.976399 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.976917 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.977961 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.978158 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.978222 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:30.981029 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:30.981030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:30.981031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:30.981039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:30.983078 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:30.983839 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:30.983841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:30.983841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:30.984440 (+ 599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:30.984875 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:30.984876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:30.984877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:30.985054 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:30.985125 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:30.985495 (+ 370us) server_negotiation.cc:299] Negotiation successful
0811 02:41:30.985609 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 02:41:30.985924 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:30.974444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38573 (local address 127.2.251.2:36297)
0811 02:41:30.974632 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:30.974676 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:30.975236 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:30.975501 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:30.975505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:30.976089 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:30.976290 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:30.976295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.977024 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.977026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:30.977871 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:30.977874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:30.978261 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:30.978262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:30.978399 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:30.978983 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:30.978993 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:30.980872 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:30.983184 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:30.983197 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:30.983200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:30.983559 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:30.984553 (+ 994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:30.984554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:30.984554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:30.984788 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:30.985149 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:30.985157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:30.985343 (+ 186us) client_negotiation.cc:769] Sending connection context
0811 02:41:30.985454 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 02:41:30.985633 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260811 02:41:30.986544 3457 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38573
I20260811 02:41:30.986666 3457 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:30.986888 3457 heartbeater.cc:511] Master 127.2.251.62:38573 requested a full tablet report, sending...
I20260811 02:41:30.987416 2958 ts_manager.cc:194] Registered new tserver with Master: 0b83d7d25d2a4f84bb7a86b87dac1495 (127.2.251.2:44047)
I20260811 02:41:30.987972 2958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.2@KRBTEST.COM'} at 127.2.251.2:36297
WARNING: no policy specified for kudu/127.2.251.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
Entry for principal kudu/127.2.251.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab.
I20260811 02:41:31.013404 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38573
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:31.109339 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:31.109580 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:31.109616 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:31.110931 3466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:31.110992 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:31.111034 3466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:31.112695 3466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.3.keytab
--principal=kudu/127.2.251.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:31.113071 3466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:31.113343 3466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:31.118662 3477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.118731 3479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.118665 3481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.118957 3466 server_base.cc:1034] running on GCE node
I20260811 02:41:31.119177 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:31.119444 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:31.120626 3466 hybrid_clock.cc:648] HybridClock initialized: now 1786416091120602 us; error 40 us; skew 500 ppm
Aug 11 02:41:31 dist-test-slave-cvwc krb5kdc[2894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:31.123487 3466 init.cc:377] Logged in from keytab as kudu/127.2.251.3@KRBTEST.COM (short username kudu)
I20260811 02:41:31.124006 3466 webserver.cc:460] Webserver started at http://127.2.251.3:42131/ using document root <none> and password file <none>
I20260811 02:41:31.124343 3466 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:31.124423 3466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:31.124589 3466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:31.126569 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "79b9c6a61315460eb18324be5ee50aed"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "7a53d8dfec3e1aef781a757f40d4e333"
server_key_iv: "85bab4d0e2cab4de613752b7e8962ccd"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.127054 3466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "79b9c6a61315460eb18324be5ee50aed"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "7a53d8dfec3e1aef781a757f40d4e333"
server_key_iv: "85bab4d0e2cab4de613752b7e8962ccd"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.128507 3466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:31.129405 3492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.129822 3466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:31.129993 3466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "79b9c6a61315460eb18324be5ee50aed"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "7a53d8dfec3e1aef781a757f40d4e333"
server_key_iv: "85bab4d0e2cab4de613752b7e8962ccd"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.130101 3466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:31.147354 3466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:31.150014 3466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:31.150159 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:31.150391 3466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:31.150717 3466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:31.150751 3466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.150781 3466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:31.150806 3466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.156776 3466 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:45743
I20260811 02:41:31.156843 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:45743 every 8 connection(s)
I20260811 02:41:31.157174 3466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:31.160311 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3466
I20260811 02:41:31.160405 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:41:31.160575 3052 external_mini_cluster.cc:1428] Setting key 5079f2f5c61430c552305f556afec919
Aug 11 02:41:31 dist-test-slave-cvwc krb5kdc[2894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416091, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.3@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:31.167176 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.158135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:50801 (local address 127.2.251.62:38573)
0811 02:41:31.158186 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.158189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.158265 (+ 76us) server_negotiation.cc:407] Connection header received
0811 02:41:31.158803 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.158804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.158817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.158885 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:31.159903 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.160549 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.161833 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.162043 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.162101 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.164490 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:31.164492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:31.164492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:31.164500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:31.165810 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:31.166204 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:31.166205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:31.166206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:31.166261 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:31.166520 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:31.166521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:31.166521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:31.166647 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:31.166704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.166987 (+ 283us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.167078 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260811 02:41:31.167372 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.157972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:38573 (local address 127.2.251.3:50801)
0811 02:41:31.158145 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.158189 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.158693 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.158978 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.158981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.159587 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:31.159798 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.159803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.160646 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.160647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.161752 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.161756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.162116 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.162117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.162258 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.162774 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:41:31.162783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:31.164372 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:31.165894 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:31.165903 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:31.165906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:31.166133 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:31.166317 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:41:31.166318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:41:31.166318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:41:31.166458 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:41:31.166703 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:31.166711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:31.166862 (+ 151us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.166955 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.167136 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260811 02:41:31.168443 3668 heartbeater.cc:347] Connected to a master server at 127.2.251.62:38573
I20260811 02:41:31.168620 3668 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:31.169528 3668 heartbeater.cc:511] Master 127.2.251.62:38573 requested a full tablet report, sending...
I20260811 02:41:31.170373 2958 ts_manager.cc:194] Registered new tserver with Master: 79b9c6a61315460eb18324be5ee50aed (127.2.251.3:45743)
I20260811 02:41:31.170846 2958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.251.3@KRBTEST.COM'} at 127.2.251.3:50801
I20260811 02:41:31.172533 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:41:31.180832 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.174044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.2.251.62:38573)
0811 02:41:31.174133 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.174135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.174353 (+ 218us) server_negotiation.cc:407] Connection header received
0811 02:41:31.174403 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.174404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.174415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.174457 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:31.175090 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.175597 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.176360 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.176553 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.176616 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.177848 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:31.177850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:31.177850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:31.177857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:31.179242 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:31.179703 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:31.179704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:31.179705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:31.179789 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:31.180074 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:31.180075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:31.180075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:31.180267 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:31.180374 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.180601 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.180712 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 02:41:31.182842 2958 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:41:31.184962 2958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:41:31.192309 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.188224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42527 (local address 127.0.0.1:42612)
0811 02:41:31.188469 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.188479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.188520 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.188996 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.188997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.188999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:31.189050 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.189052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.190078 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.190079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.191339 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.191342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.191999 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.192000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.192080 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.192095 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.192158 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.192219 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":146,"threads_started":1}
I20260811 02:41:31.192328 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.188562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.2.251.1:42527)
0811 02:41:31.188793 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.188798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.188806 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:31.188838 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.188839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.188851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.188995 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:31.189114 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.189969 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.191416 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.191916 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.191981 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.192048 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.192181 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.192233 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260811 02:41:31.192548 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.187534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:45743 (local address 127.0.0.1:49064)
0811 02:41:31.187660 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.187692 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.187765 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.188479 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.188482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.188484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:31.188562 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.188564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.190487 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.190488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.191387 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.191389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.192313 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.192314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.192376 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.192381 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.192411 (+ 30us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.192471 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260811 02:41:31.192948 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.187663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.2.251.3:45743)
0811 02:41:31.188184 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.188189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.188201 (+ 12us) server_negotiation.cc:407] Connection header received
0811 02:41:31.188238 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.188240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.188256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.188357 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:31.188633 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.190352 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.191507 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.192221 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.192728 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.192797 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.192824 (+ 27us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.192868 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":120,"threads_started":1}
I20260811 02:41:31.193127 3160 tablet_service.cc:1467] Processing CreateTablet for tablet cbf45f6d0b7a4ca5b978e1eacf541041 (DEFAULT_TABLE table=test-table [id=0db33fa8803842178a13c95b9c234458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:31.193555 3160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf45f6d0b7a4ca5b978e1eacf541041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:31.193612 3561 tablet_service.cc:1467] Processing CreateTablet for tablet cbf45f6d0b7a4ca5b978e1eacf541041 (DEFAULT_TABLE table=test-table [id=0db33fa8803842178a13c95b9c234458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:31.193981 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf45f6d0b7a4ca5b978e1eacf541041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:31.194018 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.187762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:44047 (local address 127.0.0.1:58296)
0811 02:41:31.188423 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.188434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.189176 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.189826 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.189827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.189831 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:31.189874 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.189876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.191155 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.191156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.192207 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.192210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.193708 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.193709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.193817 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.193821 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.193892 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.193947 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":71,"threads_started":1}
I20260811 02:41:31.194216 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.187951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.2.251.2:44047)
0811 02:41:31.188297 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.188302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.188777 (+ 475us) server_negotiation.cc:407] Connection header received
0811 02:41:31.189698 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.189702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.189744 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.189825 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:31.189931 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.191043 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.192311 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.193556 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.193597 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.193645 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.193916 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.193964 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260811 02:41:31.197252 3696 tablet_bootstrap.cc:492] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: Bootstrap starting.
I20260811 02:41:31.197906 3694 tablet_bootstrap.cc:492] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Bootstrap starting.
I20260811 02:41:31.198062 3696 tablet_bootstrap.cc:654] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:31.197949 3361 tablet_service.cc:1467] Processing CreateTablet for tablet cbf45f6d0b7a4ca5b978e1eacf541041 (DEFAULT_TABLE table=test-table [id=0db33fa8803842178a13c95b9c234458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:31.198297 3361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf45f6d0b7a4ca5b978e1eacf541041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:31.198654 3694 tablet_bootstrap.cc:654] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:31.198709 3696 log.cc:824] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:31.199288 3694 log.cc:824] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:31.199832 3696 tablet_bootstrap.cc:492] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: No bootstrap required, opened a new log
I20260811 02:41:31.199893 3696 ts_tablet_manager.cc:1397] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:41:31.201344 3700 tablet_bootstrap.cc:492] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Bootstrap starting.
I20260811 02:41:31.201413 3696 raft_consensus.cc:348] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.201582 3696 raft_consensus.cc:374] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:31.201627 3696 raft_consensus.cc:729] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806f980071d84cd1bfd84965385436f5, State: Initialized, Role: FOLLOWER
I20260811 02:41:31.201785 3696 consensus_queue.cc:260] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.202042 3700 tablet_bootstrap.cc:654] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:31.202088 3696 ts_tablet_manager.cc:1428] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 02:41:31.202231 3258 heartbeater.cc:503] Master 127.2.251.62:38573 was elected leader, sending a full tablet report...
I20260811 02:41:31.202471 3694 tablet_bootstrap.cc:492] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: No bootstrap required, opened a new log
I20260811 02:41:31.202548 3694 ts_tablet_manager.cc:1397] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 02:41:31.202564 3700 log.cc:824] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:31.203258 3700 tablet_bootstrap.cc:492] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: No bootstrap required, opened a new log
I20260811 02:41:31.203318 3700 ts_tablet_manager.cc:1397] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 02:41:31.204341 3694 raft_consensus.cc:348] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.204493 3694 raft_consensus.cc:374] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:31.204514 3694 raft_consensus.cc:729] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b9c6a61315460eb18324be5ee50aed, State: Initialized, Role: FOLLOWER
I20260811 02:41:31.204615 3694 consensus_queue.cc:260] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.204916 3694 ts_tablet_manager.cc:1428] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 02:41:31.204943 3668 heartbeater.cc:503] Master 127.2.251.62:38573 was elected leader, sending a full tablet report...
I20260811 02:41:31.204984 3700 raft_consensus.cc:348] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.205137 3700 raft_consensus.cc:374] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:31.205168 3700 raft_consensus.cc:729] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b83d7d25d2a4f84bb7a86b87dac1495, State: Initialized, Role: FOLLOWER
I20260811 02:41:31.205278 3700 consensus_queue.cc:260] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.205564 3700 ts_tablet_manager.cc:1428] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260811 02:41:31.205580 3457 heartbeater.cc:503] Master 127.2.251.62:38573 was elected leader, sending a full tablet report...
W20260811 02:41:31.224334 3458 tablet.cc:2369] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 02:41:31.305951 3261 tablet.cc:2369] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 02:41:31.313019 3704 raft_consensus.cc:482] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:31.313163 3704 raft_consensus.cc:504] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.313556 3704 leader_election.cc:288] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b9c6a61315460eb18324be5ee50aed (127.2.251.3:45743), 806f980071d84cd1bfd84965385436f5 (127.2.251.1:42527)
I20260811 02:41:31.317024 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.313908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:33691 (local address 127.2.251.3:45743)
0811 02:41:31.313959 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.313961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.313972 (+ 11us) server_negotiation.cc:407] Connection header received
0811 02:41:31.314003 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.314004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.314014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.314081 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:31.314319 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.314979 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.316077 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.316676 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.316732 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.316744 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.316933 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.316966 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260811 02:41:31.317051 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.313831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:45743 (local address 127.2.251.2:33691)
0811 02:41:31.313889 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.313901 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.313951 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.314101 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.314102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.314104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:31.314181 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.314184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.315082 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.315083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.315998 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.316000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.316766 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.316767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.316857 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.316861 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.316927 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.316972 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 02:41:31.317438 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf45f6d0b7a4ca5b978e1eacf541041" candidate_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b9c6a61315460eb18324be5ee50aed" is_pre_election: true
I20260811 02:41:31.317541 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.313972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42527 (local address 127.2.251.2:46383)
0811 02:41:31.314144 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.314154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.314303 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.314504 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.314504 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.314507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:31.314584 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.314586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.315523 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.315523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.316578 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.316581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.317316 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.317317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.317395 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.317399 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.317449 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.317480 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260811 02:41:31.317571 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.314108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:46383 (local address 127.2.251.1:42527)
0811 02:41:31.314172 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.314174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.314194 (+ 20us) server_negotiation.cc:407] Connection header received
0811 02:41:31.314413 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.314414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.314426 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.314479 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:31.314746 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.315426 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.316685 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.317239 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.317302 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.317313 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.317475 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.317513 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 02:41:31.317624 3594 raft_consensus.cc:2454] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b83d7d25d2a4f84bb7a86b87dac1495 in term 0.
I20260811 02:41:31.317896 3291 leader_election.cc:302] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b83d7d25d2a4f84bb7a86b87dac1495, 79b9c6a61315460eb18324be5ee50aed; no voters:
I20260811 02:41:31.317979 3192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf45f6d0b7a4ca5b978e1eacf541041" candidate_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806f980071d84cd1bfd84965385436f5" is_pre_election: true
I20260811 02:41:31.318039 3704 raft_consensus.cc:2781] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:31.318104 3704 raft_consensus.cc:482] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:31.318135 3704 raft_consensus.cc:3037] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:31.318132 3192 raft_consensus.cc:2454] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b83d7d25d2a4f84bb7a86b87dac1495 in term 0.
I20260811 02:41:31.318903 3704 raft_consensus.cc:504] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.319018 3704 leader_election.cc:288] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [CANDIDATE]: Term 1 election: Requested vote from peers 79b9c6a61315460eb18324be5ee50aed (127.2.251.3:45743), 806f980071d84cd1bfd84965385436f5 (127.2.251.1:42527)
I20260811 02:41:31.319208 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf45f6d0b7a4ca5b978e1eacf541041" candidate_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b9c6a61315460eb18324be5ee50aed"
I20260811 02:41:31.319255 3192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf45f6d0b7a4ca5b978e1eacf541041" candidate_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806f980071d84cd1bfd84965385436f5"
I20260811 02:41:31.319303 3594 raft_consensus.cc:3037] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:31.319319 3192 raft_consensus.cc:3037] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:31.320134 3192 raft_consensus.cc:2454] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b83d7d25d2a4f84bb7a86b87dac1495 in term 1.
I20260811 02:41:31.320333 3291 leader_election.cc:302] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b83d7d25d2a4f84bb7a86b87dac1495, 806f980071d84cd1bfd84965385436f5; no voters:
I20260811 02:41:31.320351 3594 raft_consensus.cc:2454] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b83d7d25d2a4f84bb7a86b87dac1495 in term 1.
I20260811 02:41:31.320430 3704 raft_consensus.cc:2781] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:31.320592 3704 raft_consensus.cc:686] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 1 LEADER]: Becoming Leader. State: Replica: 0b83d7d25d2a4f84bb7a86b87dac1495, State: Running, Role: LEADER
I20260811 02:41:31.320691 3704 consensus_queue.cc:237] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } }
I20260811 02:41:31.321630 2958 catalog_manager.cc:5495] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b83d7d25d2a4f84bb7a86b87dac1495 (127.2.251.2). New cstate: current_term: 1 leader_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b83d7d25d2a4f84bb7a86b87dac1495" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 } health_report { overall_health: UNKNOWN } } }
I20260811 02:41:31.339368 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.336892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.2.251.2:44047)
0811 02:41:31.336942 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.336944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.336952 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:31.336991 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.336992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.337014 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.337076 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:41:31.337195 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.337694 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.338497 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.338694 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.338757 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.338979 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:41:31.339095 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:41:31.339176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.339251 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.339278 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:41:31.341464 3594 raft_consensus.cc:1261] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 1 FOLLOWER]: Refusing update from remote peer 0b83d7d25d2a4f84bb7a86b87dac1495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:31.341464 3192 raft_consensus.cc:1261] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 1 FOLLOWER]: Refusing update from remote peer 0b83d7d25d2a4f84bb7a86b87dac1495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:31.341944 3704 consensus_queue.cc:1035] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b9c6a61315460eb18324be5ee50aed" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:31.342216 3706 consensus_queue.cc:1035] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806f980071d84cd1bfd84965385436f5" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:31.344396 3710 mvcc.cc:205] Tried to move back new op lower bound from 7317160310131773440 to 7317160310050091008. Current Snapshot: MvccSnapshot[applied={T|T < 7317160310131773440}]
I20260811 02:41:31.345990 3711 mvcc.cc:205] Tried to move back new op lower bound from 7317160310131773440 to 7317160310050091008. Current Snapshot: MvccSnapshot[applied={T|T < 7317160310131773440 or (T in {7317160310131773440})}]
I20260811 02:41:31.347287 2957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:31.347402 2957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:31.348203 2957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbeaa4bd7630473780b209ac261e823a: Sending DeleteTablet for 3 replicas of tablet cbf45f6d0b7a4ca5b978e1eacf541041
I20260811 02:41:31.348554 3561 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf45f6d0b7a4ca5b978e1eacf541041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:31 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:49064
I20260811 02:41:31.348567 3361 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf45f6d0b7a4ca5b978e1eacf541041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:31 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:58296
I20260811 02:41:31.348619 3160 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf45f6d0b7a4ca5b978e1eacf541041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:31 UTC) from {username='kudu', principal='kudu/127.2.251.62@KRBTEST.COM'} at 127.0.0.1:42612
I20260811 02:41:31.348932 3726 tablet_replica.cc:330] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: stopping tablet replica
I20260811 02:41:31.348945 3725 tablet_replica.cc:330] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: stopping tablet replica
I20260811 02:41:31.349030 3726 raft_consensus.cc:2229] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:41:31.349049 3725 raft_consensus.cc:2229] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:31.349124 3726 raft_consensus.cc:2258] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:31.349249 3727 tablet_replica.cc:330] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5: stopping tablet replica
I20260811 02:41:31.349366 3727 raft_consensus.cc:2229] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:31.349409 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3061
I20260811 02:41:31.349519 3727 raft_consensus.cc:2258] T cbf45f6d0b7a4ca5b978e1eacf541041 P 806f980071d84cd1bfd84965385436f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:31.349632 3726 ts_tablet_manager.cc:1905] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:31.351399 3726 ts_tablet_manager.cc:1918] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:31.351485 3726 log.cc:1196] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/cbf45f6d0b7a4ca5b978e1eacf541041
I20260811 02:41:31.351773 3726 ts_tablet_manager.cc:1939] T cbf45f6d0b7a4ca5b978e1eacf541041 P 0b83d7d25d2a4f84bb7a86b87dac1495: Deleting consensus metadata
I20260811 02:41:31.352219 2941 catalog_manager.cc:4841] TS 0b83d7d25d2a4f84bb7a86b87dac1495 (127.2.251.2:44047): tablet cbf45f6d0b7a4ca5b978e1eacf541041 (table test-table [id=0db33fa8803842178a13c95b9c234458]) successfully deleted
I20260811 02:41:31.360548 3725 raft_consensus.cc:2258] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:31.361310 3725 ts_tablet_manager.cc:1905] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:31.363238 3725 ts_tablet_manager.cc:1918] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:31.363317 3725 log.cc:1196] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/cbf45f6d0b7a4ca5b978e1eacf541041
I20260811 02:41:31.363652 3725 ts_tablet_manager.cc:1939] T cbf45f6d0b7a4ca5b978e1eacf541041 P 79b9c6a61315460eb18324be5ee50aed: Deleting consensus metadata
I20260811 02:41:31.364171 2941 catalog_manager.cc:4841] TS 79b9c6a61315460eb18324be5ee50aed (127.2.251.3:45743): tablet cbf45f6d0b7a4ca5b978e1eacf541041 (table test-table [id=0db33fa8803842178a13c95b9c234458]) successfully deleted
W20260811 02:41:31.366405 2941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:42527 (error 108)
I20260811 02:41:31.366644 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3266
I20260811 02:41:31.379832 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.379607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:42527 (local address 127.0.0.1:42622)
0811 02:41:31.379686 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.379781 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 02:41:31.380023 2941 catalog_manager.cc:4584] TS 806f980071d84cd1bfd84965385436f5 (127.2.251.1:42527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbf45f6d0b7a4ca5b978e1eacf541041: Network error: Client connection negotiation failed: client connection to 127.2.251.1:42527: connect: Connection refused (error 111)
I20260811 02:41:31.383978 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3466
I20260811 02:41:31.396046 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 2908
2026-08-11T02:41:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2281 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T02:41:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:31Z Disabled control of system clock
I20260811 02:41:31.433624 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:41771
--webserver_interface=127.2.251.62
--webserver_port=0
--builtin_ntp_servers=127.2.251.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:41771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:31.522352 3735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:31.522616 3735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:31.522639 3735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:31.523962 3735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:31.524026 3735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:31.524044 3735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:31.524061 3735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:31.524078 3735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:31.526080 3735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:41771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:41771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:31.526367 3735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:31.526656 3735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:31.531991 3741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.531994 3735 server_base.cc:1034] running on GCE node
W20260811 02:41:31.531993 3743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.532019 3745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.532632 3735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:31.532943 3735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:31.534162 3735 hybrid_clock.cc:648] HybridClock initialized: now 1786416091534128 us; error 41 us; skew 500 ppm
I20260811 02:41:31.536015 3735 webserver.cc:460] Webserver started at http://127.2.251.62:35327/ using document root <none> and password file <none>
I20260811 02:41:31.536273 3735 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:31.536321 3735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:31.536432 3735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:31.537580 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "6c7f38fd8e92439ea81c77c2a9126f95"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "67c342776d962f36a2158ffa6f32e64e"
server_key_iv: "6ea7e38a7128f6e079e002aa55a828f2"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.537982 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "6c7f38fd8e92439ea81c77c2a9126f95"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "67c342776d962f36a2158ffa6f32e64e"
server_key_iv: "6ea7e38a7128f6e079e002aa55a828f2"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.539460 3735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:31.540424 3753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.540639 3735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:31.540728 3735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "6c7f38fd8e92439ea81c77c2a9126f95"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "67c342776d962f36a2158ffa6f32e64e"
server_key_iv: "6ea7e38a7128f6e079e002aa55a828f2"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.540814 3735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:31.552433 3735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:31.552785 3735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:31.552917 3735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:31.556988 3735 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:41771
I20260811 02:41:31.557057 3835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:41771 every 8 connection(s)
I20260811 02:41:31.557391 3735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:31.558606 3837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:31.559751 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3735
I20260811 02:41:31.559887 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:31.560107 3052 external_mini_cluster.cc:1428] Setting key 4de9685d47bc051c883fa5d04518cc64
I20260811 02:41:31.561755 3837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: Bootstrap starting.
I20260811 02:41:31.562451 3837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:31.562716 3837 log.cc:824] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:31.563403 3837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: No bootstrap required, opened a new log
I20260811 02:41:31.565344 3837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41771 } }
I20260811 02:41:31.565493 3837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:31.565528 3837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7f38fd8e92439ea81c77c2a9126f95, State: Initialized, Role: FOLLOWER
I20260811 02:41:31.565661 3837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41771 } }
I20260811 02:41:31.565754 3837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:31.565788 3837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:31.565834 3837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:31.566071 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.560740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.2.251.62:41771)
0811 02:41:31.560940 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.560948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.560981 (+ 33us) server_negotiation.cc:407] Connection header received
0811 02:41:31.561483 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.561490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.561563 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.561685 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:31.562000 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.562997 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.563856 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.564382 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.564528 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.565288 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:31.565292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:31.565293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:31.565463 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:31.565497 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:31.565508 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:31.565560 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.565784 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.565880 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260811 02:41:31.566512 3837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41771 } }
I20260811 02:41:31.566624 3837 leader_election.cc:302] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c7f38fd8e92439ea81c77c2a9126f95; no voters:
I20260811 02:41:31.566813 3837 leader_election.cc:288] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:31.566906 3844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:31.567015 3844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [term 1 LEADER]: Becoming Leader. State: Replica: 6c7f38fd8e92439ea81c77c2a9126f95, State: Running, Role: LEADER
I20260811 02:41:31.567087 3844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41771 } }
I20260811 02:41:31.567116 3837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:31.568384 3845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41771 } } }
I20260811 02:41:31.568608 3846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7f38fd8e92439ea81c77c2a9126f95. Latest consensus state: current_term: 1 leader_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7f38fd8e92439ea81c77c2a9126f95" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41771 } } }
I20260811 02:41:31.568650 3845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:31.568696 3846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95 [sys.catalog]: This master's current role is: LEADER
W20260811 02:41:31.569110 3861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:41:31.569173 3861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:41:31.569276 3862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:31.569449 3862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:31.571143 3862 catalog_manager.cc:1347] Generated new cluster ID: 52b5c341d1a7488eab4dc8ec0a4ed837
I20260811 02:41:31.571203 3862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:31.581092 3862 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:31.581625 3862 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:31.584750 3862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: Generated new TSK 0
I20260811 02:41:31.584955 3862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:31.587615 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.1:0
--local_ip_for_outbound_sockets=127.2.251.1
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41771
--builtin_ntp_servers=127.2.251.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:31.679955 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:31.680187 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:31.680210 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:31.681583 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:31.681638 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:31.681666 3873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.1
I20260811 02:41:31.683193 3873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.251.1
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.1
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:31.683481 3873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:31.683815 3873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:31.690002 3879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.690122 3881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.690200 3884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.690325 3873 server_base.cc:1034] running on GCE node
I20260811 02:41:31.690548 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:31.690793 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:31.691956 3873 hybrid_clock.cc:648] HybridClock initialized: now 1786416091691940 us; error 28 us; skew 500 ppm
I20260811 02:41:31.693434 3873 webserver.cc:460] Webserver started at http://127.2.251.1:46693/ using document root <none> and password file <none>
I20260811 02:41:31.693758 3873 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:31.693826 3873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:31.693962 3873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:31.695094 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data/instance:
uuid: "da72982d0cf94b46897a1ba0252894f8"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "e8b02df7217b20e7a5983a3de8803a57"
server_key_iv: "60bb84d8997baa7f84af6db7944f33f4"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.695465 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance:
uuid: "da72982d0cf94b46897a1ba0252894f8"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "e8b02df7217b20e7a5983a3de8803a57"
server_key_iv: "60bb84d8997baa7f84af6db7944f33f4"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.696923 3873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:31.697883 3890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.698047 3873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:31.698144 3873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal
uuid: "da72982d0cf94b46897a1ba0252894f8"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "e8b02df7217b20e7a5983a3de8803a57"
server_key_iv: "60bb84d8997baa7f84af6db7944f33f4"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.698218 3873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:31.706763 3873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:31.707130 3873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:31.707275 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:31.707532 3873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:31.707892 3873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:31.707928 3873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.707957 3873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:31.707983 3873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.714455 3873 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.1:36943
I20260811 02:41:31.714506 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.1:36943 every 8 connection(s)
I20260811 02:41:31.714875 3873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/data/info.pb
I20260811 02:41:31.721459 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.715686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:44351 (local address 127.2.251.62:41771)
0811 02:41:31.715772 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.715774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.716002 (+ 228us) server_negotiation.cc:407] Connection header received
0811 02:41:31.716564 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.716565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.716578 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.716634 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:31.717848 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.718423 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.719622 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.719839 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.719900 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.721124 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:31.721126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:31.721127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:31.721133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:31.721154 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:31.721159 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:31.721206 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.721315 (+ 109us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.721364 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 02:41:31.721699 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.715617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41771 (local address 127.2.251.1:44351)
0811 02:41:31.715865 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.715902 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.716460 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.716723 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.716726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.716735 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:31.717591 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.717606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.718543 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.718545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.719514 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.719518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.719928 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.719930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.720094 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.720904 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:31.720905 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:31.720949 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:31.720987 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:31.721218 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:31.721224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:31.721229 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.721301 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.721452 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260811 02:41:31.722613 4046 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41771
I20260811 02:41:31.722736 4046 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:31.722970 4046 heartbeater.cc:511] Master 127.2.251.62:41771 requested a full tablet report, sending...
I20260811 02:41:31.723769 3777 ts_manager.cc:194] Registered new tserver with Master: da72982d0cf94b46897a1ba0252894f8 (127.2.251.1:36943)
I20260811 02:41:31.724107 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 3873
I20260811 02:41:31.724210 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-0/wal/instance
I20260811 02:41:31.724418 3052 external_mini_cluster.cc:1428] Setting key c29a07dd0b510acd8fb21017c2aa107d
I20260811 02:41:31.724526 3777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.1:44351
I20260811 02:41:31.725955 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.2:0
--local_ip_for_outbound_sockets=127.2.251.2
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41771
--builtin_ntp_servers=127.2.251.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:31.815320 4051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:31.815551 4051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:31.815574 4051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:31.816751 4051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:31.816797 4051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:31.816824 4051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.2
I20260811 02:41:31.818269 4051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.251.2
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.2
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:31.818511 4051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:31.818813 4051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:31.824280 4068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.824280 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.824298 4064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.824435 4051 server_base.cc:1034] running on GCE node
I20260811 02:41:31.824657 4051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:31.824932 4051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:31.826089 4051 hybrid_clock.cc:648] HybridClock initialized: now 1786416091826052 us; error 84 us; skew 500 ppm
I20260811 02:41:31.827476 4051 webserver.cc:460] Webserver started at http://127.2.251.2:45419/ using document root <none> and password file <none>
I20260811 02:41:31.827744 4051 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:31.827800 4051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:31.827950 4051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:31.829078 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data/instance:
uuid: "78f55a3dcbdb44cea459c22ea48a8fb7"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "80239a873156d2252357b36f4b46f5c8"
server_key_iv: "7230a5460855849c44ad98dcd5f23b8a"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.829435 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance:
uuid: "78f55a3dcbdb44cea459c22ea48a8fb7"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "80239a873156d2252357b36f4b46f5c8"
server_key_iv: "7230a5460855849c44ad98dcd5f23b8a"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.831202 4051 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 02:41:31.832453 4076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.832785 4051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:31.832906 4051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal
uuid: "78f55a3dcbdb44cea459c22ea48a8fb7"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "80239a873156d2252357b36f4b46f5c8"
server_key_iv: "7230a5460855849c44ad98dcd5f23b8a"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.833004 4051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:31.844623 4051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:31.844995 4051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:31.845163 4051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:31.845429 4051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:31.845853 4051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:31.845894 4051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.845927 4051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:31.845952 4051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.852459 4051 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.2:35445
I20260811 02:41:31.852525 4268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.2:35445 every 8 connection(s)
I20260811 02:41:31.852901 4051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/data/info.pb
I20260811 02:41:31.859042 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.853860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.2:54299 (local address 127.2.251.62:41771)
0811 02:41:31.853932 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.853934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.854161 (+ 227us) server_negotiation.cc:407] Connection header received
0811 02:41:31.854673 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.854674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.854685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.854750 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:31.855469 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.856012 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.857203 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.857412 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.857483 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.858702 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:31.858704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:31.858704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:31.858710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:31.858726 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:31.858730 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:31.858801 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.858937 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.858966 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 02:41:31.859234 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.853805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41771 (local address 127.2.251.2:54299)
0811 02:41:31.854002 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.854044 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.854557 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.854834 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.854837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.854846 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:31.855363 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.855367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.856126 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.856127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.857096 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.857100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.857551 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.857553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.857687 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.858436 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:31.858436 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:31.858479 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:31.858575 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:31.858827 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:31.858832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:31.858835 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.858914 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.859016 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260811 02:41:31.859750 4270 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41771
I20260811 02:41:31.859856 4270 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:31.860060 4270 heartbeater.cc:511] Master 127.2.251.62:41771 requested a full tablet report, sending...
I20260811 02:41:31.860513 3777 ts_manager.cc:194] Registered new tserver with Master: 78f55a3dcbdb44cea459c22ea48a8fb7 (127.2.251.2:35445)
I20260811 02:41:31.860873 3777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.2:54299
I20260811 02:41:31.862856 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4051
I20260811 02:41:31.862953 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal/instance
I20260811 02:41:31.863144 3052 external_mini_cluster.cc:1428] Setting key aa09b0ad1b7cf80f097d9945616cdfe2
I20260811 02:41:31.864492 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.251.3:0
--local_ip_for_outbound_sockets=127.2.251.3
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41771
--builtin_ntp_servers=127.2.251.20:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 02:41:31.952934 4275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:31.953172 4275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:31.953195 4275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:31.954412 4275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:31.954460 4275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:31.954487 4275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.251.3
I20260811 02:41:31.955856 4275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.251.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.251.3
--webserver_port=0
--tserver_master_addrs=127.2.251.62:41771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.251.3
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:31.956074 4275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:31.956367 4275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:31.961509 4282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:31.961586 4286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.961650 4275 server_base.cc:1034] running on GCE node
W20260811 02:41:31.961757 4281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:31.962006 4275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:31.962252 4275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:31.963435 4275 hybrid_clock.cc:648] HybridClock initialized: now 1786416091963416 us; error 38 us; skew 500 ppm
I20260811 02:41:31.964844 4275 webserver.cc:460] Webserver started at http://127.2.251.3:35557/ using document root <none> and password file <none>
I20260811 02:41:31.965129 4275 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:31.965185 4275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:31.965317 4275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:31.966472 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data/instance:
uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "de265757e5bb3fed551abfeb8b4035b9"
server_key_iv: "e4dbcd11faebca6ba2805fc8391df86e"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.966863 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance:
uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "de265757e5bb3fed551abfeb8b4035b9"
server_key_iv: "e4dbcd11faebca6ba2805fc8391df86e"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.968266 4275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260811 02:41:31.969112 4293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.969408 4275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:31.969502 4275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal
uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa"
format_stamp: "Formatted at 2026-08-11 02:41:31 on dist-test-slave-cvwc"
server_key: "de265757e5bb3fed551abfeb8b4035b9"
server_key_iv: "e4dbcd11faebca6ba2805fc8391df86e"
server_key_version: "encryptionkey@0"
I20260811 02:41:31.969578 4275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:31.981042 4275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:31.981447 4275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:31.981588 4275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:31.981895 4275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 02:41:31.982261 4275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 02:41:31.982298 4275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.982328 4275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 02:41:31.982352 4275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:31.988176 4275 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.3:36709
I20260811 02:41:31.988233 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.3:36709 every 8 connection(s)
I20260811 02:41:31.988591 4275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/data/info.pb
I20260811 02:41:31.991649 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4275
I20260811 02:41:31.991779 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal/instance
I20260811 02:41:31.991974 3052 external_mini_cluster.cc:1428] Setting key f40c7d7dcf9115c77f3095c1a16a1f93
I20260811 02:41:31.994517 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.989581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.3:42389 (local address 127.2.251.62:41771)
0811 02:41:31.989653 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:31.989655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:31.989669 (+ 14us) server_negotiation.cc:407] Connection header received
0811 02:41:31.990403 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:31.990404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:31.990416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:31.990483 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:31.991365 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.991910 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.993169 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.993386 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.993441 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.994253 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:31.994254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:31.994255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:31.994260 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:31.994275 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:31.994280 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:31.994325 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:31.994409 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 02:41:31.994436 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 02:41:31.994722 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:31.989349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:41771 (local address 127.2.251.3:42389)
0811 02:41:31.989550 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:31.989593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:31.990256 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:31.990631 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:31.990635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:31.990656 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:41:31.991244 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.991250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.992072 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.992073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.993093 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:31.993096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:31.993462 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:31.993463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:31.993602 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:31.994116 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 02:41:31.994116 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:41:31.994146 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 02:41:31.994171 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:41:31.994331 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:41:31.994337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:41:31.994340 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:31.994405 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 02:41:31.994511 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260811 02:41:31.995151 4462 heartbeater.cc:347] Connected to a master server at 127.2.251.62:41771
I20260811 02:41:31.995245 4462 heartbeater.cc:464] Registering TS with master...
I20260811 02:41:31.995419 4462 heartbeater.cc:511] Master 127.2.251.62:41771 requested a full tablet report, sending...
I20260811 02:41:31.995805 3777 ts_manager.cc:194] Registered new tserver with Master: bbb92424da1b4e66ae2a6399b4c7b9fa (127.2.251.3:36709)
I20260811 02:41:31.996168 3777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.251.3:42389
I20260811 02:41:32.004364 3052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 02:41:32.008836 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.005800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.2.251.62:41771)
0811 02:41:32.005852 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.005853 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.006084 (+ 231us) server_negotiation.cc:407] Connection header received
0811 02:41:32.006117 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.006118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.006130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.006179 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:32.006352 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.006846 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.007577 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.007764 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.007819 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.008578 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:32.008579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:32.008580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:32.008585 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:32.008601 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:32.008605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:32.008649 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.008707 (+ 58us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.008743 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:41:32.011011 3777 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 02:41:32.012624 3777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 02:41:32.018796 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.014968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.2.251.3:36709)
0811 02:41:32.015228 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.015232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.015252 (+ 20us) server_negotiation.cc:407] Connection header received
0811 02:41:32.015290 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.015292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.015305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.015369 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:32.015526 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.016422 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.017823 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.018407 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018480 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.018578 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.018664 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.018703 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260811 02:41:32.018785 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.014787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:36709 (local address 127.0.0.1:53624)
0811 02:41:32.014982 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.015023 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:32.015089 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:32.015390 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:32.015393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:32.015395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:32.015463 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.015464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.016501 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.016502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.017682 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.017685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.018494 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.018572 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.018586 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 02:41:32.018651 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 02:41:32.018704 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
I20260811 02:41:32.019059 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.014787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:36943 (local address 127.0.0.1:59240)
0811 02:41:32.015127 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.015136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:32.015175 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:32.015609 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:32.015610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:32.015611 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:32.015642 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.015643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.016885 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.016886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.017750 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.017753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.018689 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.018754 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.018759 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 02:41:32.018809 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 02:41:32.018988 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260811 02:41:32.019167 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.014790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:35445 (local address 127.0.0.1:58790)
0811 02:41:32.015221 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.015231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:32.015290 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:32.015508 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:32.015509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:32.015512 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:32.015556 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.015558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.016869 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.016870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.017986 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.017989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.018962 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.019016 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.019020 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:32.019072 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 02:41:32.019106 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":26,"thread_start_us":136,"threads_started":1}
I20260811 02:41:32.019248 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.015063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.2.251.2:35445)
0811 02:41:32.015205 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.015210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.015277 (+ 67us) server_negotiation.cc:407] Connection header received
0811 02:41:32.015357 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.015360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.015375 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.015447 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:32.015632 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.016779 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018082 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.018773 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018844 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.018928 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.019107 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.019153 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260811 02:41:32.019479 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.014969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.2.251.1:36943)
0811 02:41:32.015403 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.015407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.015416 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:41:32.015452 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.015454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.015466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.015530 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:32.015714 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.016793 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.017811 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.018498 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.018565 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.018651 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.019285 (+ 634us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.019402 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":84,"threads_started":1}
I20260811 02:41:32.019946 4363 tablet_service.cc:1467] Processing CreateTablet for tablet 4d74e242b0114631843c0907d158450c (DEFAULT_TABLE table=test-table [id=6469f0f6bbc04496ab28d990d5ba70b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:32.020150 4154 tablet_service.cc:1467] Processing CreateTablet for tablet 4d74e242b0114631843c0907d158450c (DEFAULT_TABLE table=test-table [id=6469f0f6bbc04496ab28d990d5ba70b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:32.020536 4154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d74e242b0114631843c0907d158450c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:32.020538 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d74e242b0114631843c0907d158450c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:32.020488 3961 tablet_service.cc:1467] Processing CreateTablet for tablet 4d74e242b0114631843c0907d158450c (DEFAULT_TABLE table=test-table [id=6469f0f6bbc04496ab28d990d5ba70b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 02:41:32.020812 3961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d74e242b0114631843c0907d158450c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:32.024338 4494 tablet_bootstrap.cc:492] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Bootstrap starting.
I20260811 02:41:32.024540 4492 tablet_bootstrap.cc:492] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Bootstrap starting.
I20260811 02:41:32.025094 4494 tablet_bootstrap.cc:654] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:32.025444 4494 log.cc:824] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:32.025607 4492 tablet_bootstrap.cc:654] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:32.025835 4491 tablet_bootstrap.cc:492] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: Bootstrap starting.
I20260811 02:41:32.026412 4491 tablet_bootstrap.cc:654] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:32.026687 4491 log.cc:824] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:32.026708 4492 log.cc:824] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:32.027086 4494 tablet_bootstrap.cc:492] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: No bootstrap required, opened a new log
I20260811 02:41:32.027150 4494 ts_tablet_manager.cc:1397] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 02:41:32.027590 4492 tablet_bootstrap.cc:492] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: No bootstrap required, opened a new log
I20260811 02:41:32.027652 4492 ts_tablet_manager.cc:1397] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:41:32.027662 4491 tablet_bootstrap.cc:492] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: No bootstrap required, opened a new log
I20260811 02:41:32.027724 4491 ts_tablet_manager.cc:1397] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260811 02:41:32.029925 4492 raft_consensus.cc:348] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.029943 4494 raft_consensus.cc:348] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.030094 4492 raft_consensus.cc:374] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:32.030120 4494 raft_consensus.cc:374] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:32.030129 4492 raft_consensus.cc:729] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f55a3dcbdb44cea459c22ea48a8fb7, State: Initialized, Role: FOLLOWER
I20260811 02:41:32.030164 4494 raft_consensus.cc:729] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb92424da1b4e66ae2a6399b4c7b9fa, State: Initialized, Role: FOLLOWER
I20260811 02:41:32.030280 4492 consensus_queue.cc:260] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.030380 4494 consensus_queue.cc:260] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.030599 4492 ts_tablet_manager.cc:1428] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 02:41:32.030725 4270 heartbeater.cc:503] Master 127.2.251.62:41771 was elected leader, sending a full tablet report...
I20260811 02:41:32.030715 4494 ts_tablet_manager.cc:1428] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 02:41:32.030841 4462 heartbeater.cc:503] Master 127.2.251.62:41771 was elected leader, sending a full tablet report...
I20260811 02:41:32.030857 4491 raft_consensus.cc:348] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.031039 4491 raft_consensus.cc:374] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:32.031620 4491 raft_consensus.cc:729] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da72982d0cf94b46897a1ba0252894f8, State: Initialized, Role: FOLLOWER
I20260811 02:41:32.031754 4491 consensus_queue.cc:260] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.032071 4491 ts_tablet_manager.cc:1428] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 02:41:32.032102 4046 heartbeater.cc:503] Master 127.2.251.62:41771 was elected leader, sending a full tablet report...
I20260811 02:41:32.076122 4500 raft_consensus.cc:482] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 02:41:32.076288 4500 raft_consensus.cc:504] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.076671 4500 leader_election.cc:288] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb92424da1b4e66ae2a6399b4c7b9fa (127.2.251.3:36709), 78f55a3dcbdb44cea459c22ea48a8fb7 (127.2.251.2:35445)
I20260811 02:41:32.079339 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.076896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:57751 (local address 127.2.251.3:36709)
0811 02:41:32.076952 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.076954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.076962 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:32.077014 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.077014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.077023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.077069 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:32.077201 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.077688 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.078678 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.079037 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.079097 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.079108 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.079229 (+ 121us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.079266 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 02:41:32.079340 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.076831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.3:36709 (local address 127.2.251.1:57751)
0811 02:41:32.076885 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.076898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:32.076948 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:32.077073 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:32.077074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:32.077075 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:32.077147 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.077148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.077803 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.077805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.078590 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.078593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.079108 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.079109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.079171 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.079175 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 02:41:32.079219 (+ 44us) client_negotiation.cc:240] Negotiation successful
0811 02:41:32.079262 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":26}
I20260811 02:41:32.079408 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.076932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.251.1:36929 (local address 127.2.251.2:35445)
0811 02:41:32.077006 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.077008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.077137 (+ 129us) server_negotiation.cc:407] Connection header received
0811 02:41:32.077190 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.077191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.077200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.077244 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 02:41:32.077378 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.077859 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.078742 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.079103 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.079163 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.079173 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.079303 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.079338 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 02:41:32.079653 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.076835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.2:35445 (local address 127.2.251.1:36929)
0811 02:41:32.077070 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.077078 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0811 02:41:32.077135 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:41:32.077250 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:41:32.077251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:41:32.077253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 02:41:32.077324 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.077325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.077924 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.077924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.078659 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:41:32.078662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.079174 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.079175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:41:32.079244 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.079247 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 02:41:32.079290 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 02:41:32.079331 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":42,"thread_start_us":121,"threads_started":1}
I20260811 02:41:32.079741 4393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74e242b0114631843c0907d158450c" candidate_uuid: "da72982d0cf94b46897a1ba0252894f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" is_pre_election: true
I20260811 02:41:32.079924 4393 raft_consensus.cc:2454] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da72982d0cf94b46897a1ba0252894f8 in term 0.
I20260811 02:41:32.079988 4181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74e242b0114631843c0907d158450c" candidate_uuid: "da72982d0cf94b46897a1ba0252894f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" is_pre_election: true
I20260811 02:41:32.080246 4181 raft_consensus.cc:2454] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da72982d0cf94b46897a1ba0252894f8 in term 0.
I20260811 02:41:32.080331 3897 leader_election.cc:302] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb92424da1b4e66ae2a6399b4c7b9fa, da72982d0cf94b46897a1ba0252894f8; no voters:
I20260811 02:41:32.080475 4500 raft_consensus.cc:2781] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 02:41:32.080543 4500 raft_consensus.cc:482] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 02:41:32.080569 4500 raft_consensus.cc:3037] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:32.081298 4500 raft_consensus.cc:504] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.081432 4500 leader_election.cc:288] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [CANDIDATE]: Term 1 election: Requested vote from peers bbb92424da1b4e66ae2a6399b4c7b9fa (127.2.251.3:36709), 78f55a3dcbdb44cea459c22ea48a8fb7 (127.2.251.2:35445)
I20260811 02:41:32.081583 4393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74e242b0114631843c0907d158450c" candidate_uuid: "da72982d0cf94b46897a1ba0252894f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa"
I20260811 02:41:32.081593 4181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d74e242b0114631843c0907d158450c" candidate_uuid: "da72982d0cf94b46897a1ba0252894f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7"
I20260811 02:41:32.081658 4393 raft_consensus.cc:3037] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:32.081662 4181 raft_consensus.cc:3037] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:32.082376 4393 raft_consensus.cc:2454] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da72982d0cf94b46897a1ba0252894f8 in term 1.
I20260811 02:41:32.082542 3897 leader_election.cc:302] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb92424da1b4e66ae2a6399b4c7b9fa, da72982d0cf94b46897a1ba0252894f8; no voters:
I20260811 02:41:32.082641 4181 raft_consensus.cc:2454] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da72982d0cf94b46897a1ba0252894f8 in term 1.
I20260811 02:41:32.082669 4500 raft_consensus.cc:2781] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:32.082847 4500 raft_consensus.cc:686] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 1 LEADER]: Becoming Leader. State: Replica: da72982d0cf94b46897a1ba0252894f8, State: Running, Role: LEADER
I20260811 02:41:32.082952 4500 consensus_queue.cc:237] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } }
I20260811 02:41:32.083851 3776 catalog_manager.cc:5495] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to da72982d0cf94b46897a1ba0252894f8 (127.2.251.1). New cstate: current_term: 1 leader_uuid: "da72982d0cf94b46897a1ba0252894f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da72982d0cf94b46897a1ba0252894f8" member_type: VOTER last_known_addr { host: "127.2.251.1" port: 36943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 } health_report { overall_health: UNKNOWN } } }
I20260811 02:41:32.095779 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.093455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.2.251.1:36943)
0811 02:41:32.093518 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:32.093520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:32.093528 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:41:32.093570 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:32.093571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:32.093592 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:32.093653 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:41:32.093811 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.094287 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.095033 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:32.095226 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:32.095284 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:32.095438 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:41:32.095551 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:41:32.095627 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:32.095648 (+ 21us) server_negotiation.cc:299] Negotiation successful
0811 02:41:32.095671 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 02:41:32.097681 4181 raft_consensus.cc:1261] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 1 FOLLOWER]: Refusing update from remote peer da72982d0cf94b46897a1ba0252894f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:32.097682 4393 raft_consensus.cc:1261] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 1 FOLLOWER]: Refusing update from remote peer da72982d0cf94b46897a1ba0252894f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:41:32.098085 4500 consensus_queue.cc:1035] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f55a3dcbdb44cea459c22ea48a8fb7" member_type: VOTER last_known_addr { host: "127.2.251.2" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:32.098261 4503 consensus_queue.cc:1035] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb92424da1b4e66ae2a6399b4c7b9fa" member_type: VOTER last_known_addr { host: "127.2.251.3" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:41:32.100854 4511 mvcc.cc:205] Tried to move back new op lower bound from 7317160313229320192 to 7317160313172393984. Current Snapshot: MvccSnapshot[applied={T|T < 7317160313229320192}]
I20260811 02:41:32.101346 4516 mvcc.cc:205] Tried to move back new op lower bound from 7317160313229320192 to 7317160313172393984. Current Snapshot: MvccSnapshot[applied={T|T < 7317160313229320192}]
I20260811 02:41:32.101464 4512 mvcc.cc:205] Tried to move back new op lower bound from 7317160313229320192 to 7317160313172393984. Current Snapshot: MvccSnapshot[applied={T|T < 7317160313229320192}]
I20260811 02:41:32.104831 3776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:32.104949 3776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 02:41:32.105760 3776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c7f38fd8e92439ea81c77c2a9126f95: Sending DeleteTablet for 3 replicas of tablet 4d74e242b0114631843c0907d158450c
I20260811 02:41:32.106101 3961 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d74e242b0114631843c0907d158450c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:32 UTC) from {username='slave'} at 127.0.0.1:59240
I20260811 02:41:32.106129 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d74e242b0114631843c0907d158450c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:32 UTC) from {username='slave'} at 127.0.0.1:53624
I20260811 02:41:32.106249 4154 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d74e242b0114631843c0907d158450c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 02:41:32 UTC) from {username='slave'} at 127.0.0.1:58790
I20260811 02:41:32.106395 4527 tablet_replica.cc:330] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8: stopping tablet replica
I20260811 02:41:32.106518 4527 raft_consensus.cc:2229] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 02:41:32.106537 4532 tablet_replica.cc:330] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: stopping tablet replica
I20260811 02:41:32.106627 4527 raft_consensus.cc:2258] T 4d74e242b0114631843c0907d158450c P da72982d0cf94b46897a1ba0252894f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:32.106755 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3873
I20260811 02:41:32.106760 4532 raft_consensus.cc:2229] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:32.106940 4532 raft_consensus.cc:2258] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:32.108006 4532 ts_tablet_manager.cc:1905] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:32.109879 4532 ts_tablet_manager.cc:1918] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:32.109956 4532 log.cc:1196] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-1/wal/wals/4d74e242b0114631843c0907d158450c
I20260811 02:41:32.110658 4532 ts_tablet_manager.cc:1939] T 4d74e242b0114631843c0907d158450c P 78f55a3dcbdb44cea459c22ea48a8fb7: Deleting consensus metadata
I20260811 02:41:32.111714 3754 catalog_manager.cc:4841] TS 78f55a3dcbdb44cea459c22ea48a8fb7 (127.2.251.2:35445): tablet 4d74e242b0114631843c0907d158450c (table test-table [id=6469f0f6bbc04496ab28d990d5ba70b8]) successfully deleted
I20260811 02:41:32.117414 4530 tablet_replica.cc:330] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: stopping tablet replica
I20260811 02:41:32.117586 4530 raft_consensus.cc:2229] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 02:41:32.117833 4530 raft_consensus.cc:2258] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 02:41:32.118386 4530 ts_tablet_manager.cc:1905] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 02:41:32.120124 4530 ts_tablet_manager.cc:1918] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 02:41:32.120209 4530 log.cc:1196] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Deleting WAL directory at /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786415831315317-3052-0/minicluster-data/ts-2/wal/wals/4d74e242b0114631843c0907d158450c
I20260811 02:41:32.120507 4530 ts_tablet_manager.cc:1939] T 4d74e242b0114631843c0907d158450c P bbb92424da1b4e66ae2a6399b4c7b9fa: Deleting consensus metadata
I20260811 02:41:32.121026 3757 catalog_manager.cc:4841] TS bbb92424da1b4e66ae2a6399b4c7b9fa (127.2.251.3:36709): tablet 4d74e242b0114631843c0907d158450c (table test-table [id=6469f0f6bbc04496ab28d990d5ba70b8]) successfully deleted
W20260811 02:41:32.122714 3755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.251.1:36943 (error 108)
I20260811 02:41:32.122951 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4051
I20260811 02:41:32.125941 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.123093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:36943 (local address 127.0.0.1:59250)
0811 02:41:32.125828 (+ 2735us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.125902 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:36943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2700}
W20260811 02:41:32.128405 3755 catalog_manager.cc:4584] TS da72982d0cf94b46897a1ba0252894f8 (127.2.251.1:36943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d74e242b0114631843c0907d158450c: Network error: Client connection negotiation failed: client connection to 127.2.251.1:36943: connect: Connection refused (error 111)
I20260811 02:41:32.135646 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4275
I20260811 02:41:32.184019 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.183841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:36943 (local address 127.0.0.1:59256)
0811 02:41:32.183930 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.183967 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:36943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 02:41:32.184448 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:32.184342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.1:36943 (local address 127.0.0.1:59264)
0811 02:41:32.184394 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0811 02:41:32.184425 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.1:36943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 02:41:32.188140 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 3735
2026-08-11T02:41:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (790 ms)
[----------] 6 tests from EncryptionPolicyTest (9812 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:32 dist-test-slave-cvwc krb5kdc[4538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:32 dist-test-slave-cvwc krb5kdc[4538](info): set up 2 sockets
Aug 11 02:41:32 dist-test-slave-cvwc krb5kdc[4538](info): commencing operation
krb5kdc: starting...
W20260811 02:41:33.385392 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:33 dist-test-slave-cvwc krb5kdc[4538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:33.516731 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:33.601346 4556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:33.601583 4556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:33.601606 4556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:33.602818 4556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:33.602866 4556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:33.602880 4556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:33.602890 4556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:33.602901 4556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:33.604442 4556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:33.604698 4556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:33.604939 4556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:33.610291 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:33.610291 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:33.610507 4556 server_base.cc:1034] running on GCE node
W20260811 02:41:33.610623 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:33.610937 4556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:33.611256 4556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:33.612509 4556 hybrid_clock.cc:648] HybridClock initialized: now 1786416093612474 us; error 32 us; skew 500 ppm
Aug 11 02:41:33 dist-test-slave-cvwc krb5kdc[4538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:33.615689 4556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:33.616210 4556 webserver.cc:460] Webserver started at http://127.0.0.1:33261/ using document root <none> and password file <none>
I20260811 02:41:33.616451 4556 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:33.616503 4556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:33.616636 4556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:33.617835 4556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "6aba5698627443e3a6ee059e53d7f51a"
format_stamp: "Formatted at 2026-08-11 02:41:33 on dist-test-slave-cvwc"
server_key: "a5f2933e84a2c8192d81b3d52ecf3846"
server_key_iv: "b295f2b796940329b71863fdefd8600e"
server_key_version: "encryptionkey@0"
I20260811 02:41:33.618211 4556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "6aba5698627443e3a6ee059e53d7f51a"
format_stamp: "Formatted at 2026-08-11 02:41:33 on dist-test-slave-cvwc"
server_key: "a5f2933e84a2c8192d81b3d52ecf3846"
server_key_iv: "b295f2b796940329b71863fdefd8600e"
server_key_version: "encryptionkey@0"
I20260811 02:41:33.619766 4556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:33.620862 4575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:33.621181 4556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:33.621297 4556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "6aba5698627443e3a6ee059e53d7f51a"
format_stamp: "Formatted at 2026-08-11 02:41:33 on dist-test-slave-cvwc"
server_key: "a5f2933e84a2c8192d81b3d52ecf3846"
server_key_iv: "b295f2b796940329b71863fdefd8600e"
server_key_version: "encryptionkey@0"
I20260811 02:41:33.621374 4556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:33.635178 4556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:33.637351 4556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:33.637508 4556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:33.641801 4556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35823
I20260811 02:41:33.641882 4680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35823 every 8 connection(s)
I20260811 02:41:33.642208 4556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:33.642769 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4556
I20260811 02:41:33.642879 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:33.643082 3052 external_mini_cluster.cc:1428] Setting key 8fd8b914ae88e23307ab99ff04e5126c
I20260811 02:41:33.643285 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:33.646641 4681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a: Bootstrap starting.
I20260811 02:41:33.647370 4681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:33.647658 4681 log.cc:824] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:33.648619 4681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a: No bootstrap required, opened a new log
Aug 11 02:41:33 dist-test-slave-cvwc krb5kdc[4538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 02:41:33.650754 4681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba5698627443e3a6ee059e53d7f51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35823 } }
I20260811 02:41:33.650942 4681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:33.650993 4681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aba5698627443e3a6ee059e53d7f51a, State: Initialized, Role: FOLLOWER
I20260811 02:41:33.651126 4681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba5698627443e3a6ee059e53d7f51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35823 } }
I20260811 02:41:33.651201 4681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:33.651243 4681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:33.651294 4681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:33.651854 4681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba5698627443e3a6ee059e53d7f51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35823 } }
I20260811 02:41:33.651945 4681 leader_election.cc:302] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aba5698627443e3a6ee059e53d7f51a; no voters:
I20260811 02:41:33.652122 4681 leader_election.cc:288] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:33.652204 4686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:33.652381 4686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [term 1 LEADER]: Becoming Leader. State: Replica: 6aba5698627443e3a6ee059e53d7f51a, State: Running, Role: LEADER
I20260811 02:41:33.652410 4681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:33.652482 4686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba5698627443e3a6ee059e53d7f51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35823 } }
I20260811 02:41:33.652853 4691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aba5698627443e3a6ee059e53d7f51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba5698627443e3a6ee059e53d7f51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35823 } } }
I20260811 02:41:33.652945 4691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:33.652865 4688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aba5698627443e3a6ee059e53d7f51a. Latest consensus state: current_term: 1 leader_uuid: "6aba5698627443e3a6ee059e53d7f51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba5698627443e3a6ee059e53d7f51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35823 } } }
I20260811 02:41:33.653157 4688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:33.653245 4702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:33.653851 4702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:33.654868 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:33.643609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59285 (local address 127.0.0.1:35823)
0811 02:41:33.643777 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:33.643784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:33.643811 (+ 27us) server_negotiation.cc:407] Connection header received
0811 02:41:33.644433 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:33.644440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:33.644498 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:33.644614 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:33.645465 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:33.646469 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:33.647481 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:33.648222 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:33.648337 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:33.650941 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:33.650945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:33.650945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:33.650956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:33.652962 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:33.653484 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:33.653488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:33.653488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:33.653607 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:33.653898 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:33.653899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:33.653899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:33.654237 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:33.654406 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:33.654559 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 02:41:33.654663 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260811 02:41:33.655611 4702 catalog_manager.cc:1347] Generated new cluster ID: f0151773c5604e1a843016b26d2d8ad0
I20260811 02:41:33.655665 4702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:33.655810 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:33.673956 4702 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:33.674667 4702 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:33.677985 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:33.670207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34259 (local address 127.0.0.1:35823)
0811 02:41:33.670262 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:33.670264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:33.670375 (+ 111us) server_negotiation.cc:407] Connection header received
0811 02:41:33.670460 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:33.670461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:33.670475 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:33.670533 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:33.671218 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:33.672068 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:33.673100 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:33.673324 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:33.673382 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:33.675175 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:33.675176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:33.675177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:33.675184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:33.676721 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:33.677112 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:33.677113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:33.677113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:33.677163 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:33.677364 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:33.677365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:33.677365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:33.677478 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:33.677564 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:33.677831 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 02:41:33.677899 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 02:41:33.678681 4702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aba5698627443e3a6ee059e53d7f51a: Generated new TSK 0
I20260811 02:41:33.678849 4702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:33.685410 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4556
W20260811 02:41:33.703985 4534 connection.cc:535] client connection to 127.0.0.1:35823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35823): Connection reset by peer (error 104)
2026-08-11T02:41:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1502 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:33 dist-test-slave-cvwc krb5kdc[4732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:33 dist-test-slave-cvwc krb5kdc[4732](info): set up 2 sockets
Aug 11 02:41:33 dist-test-slave-cvwc krb5kdc[4732](info): commencing operation
krb5kdc: starting...
W20260811 02:41:34.922534 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:35 dist-test-slave-cvwc krb5kdc[4732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:35.057765 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:35.151953 4751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:35.152218 4751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:35.152254 4751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:35.153576 4751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:35.153640 4751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:35.153659 4751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:35.153676 4751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:35.153689 4751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:35.155534 4751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:35.155854 4751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:35.156106 4751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:35.161938 4760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:35.161938 4764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:35.161959 4751 server_base.cc:1034] running on GCE node
W20260811 02:41:35.161938 4757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:35.162498 4751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:35.162837 4751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:35.164045 4751 hybrid_clock.cc:648] HybridClock initialized: now 1786416095164015 us; error 49 us; skew 500 ppm
Aug 11 02:41:35 dist-test-slave-cvwc krb5kdc[4732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:35.172115 4751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:35.172665 4751 webserver.cc:460] Webserver started at http://127.0.0.1:41671/ using document root <none> and password file <none>
I20260811 02:41:35.172936 4751 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:35.172986 4751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:35.173112 4751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:35.174338 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "7c7934b454d34f36b156178687915437"
format_stamp: "Formatted at 2026-08-11 02:41:35 on dist-test-slave-cvwc"
server_key: "b97d08b6134d2f33ae4c13bea3e96a1b"
server_key_iv: "ea7830b6153aa84dd9c580a46498d630"
server_key_version: "encryptionkey@0"
I20260811 02:41:35.174691 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "7c7934b454d34f36b156178687915437"
format_stamp: "Formatted at 2026-08-11 02:41:35 on dist-test-slave-cvwc"
server_key: "b97d08b6134d2f33ae4c13bea3e96a1b"
server_key_iv: "ea7830b6153aa84dd9c580a46498d630"
server_key_version: "encryptionkey@0"
I20260811 02:41:35.176321 4751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:35.177358 4774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:35.177704 4751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:35.177873 4751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "7c7934b454d34f36b156178687915437"
format_stamp: "Formatted at 2026-08-11 02:41:35 on dist-test-slave-cvwc"
server_key: "b97d08b6134d2f33ae4c13bea3e96a1b"
server_key_iv: "ea7830b6153aa84dd9c580a46498d630"
server_key_version: "encryptionkey@0"
I20260811 02:41:35.177956 4751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:35.200353 4751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:35.203732 4751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:35.203938 4751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:35.211407 4751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40259
I20260811 02:41:35.211515 4859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40259 every 8 connection(s)
I20260811 02:41:35.212042 4751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:35.213317 4860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:35.215147 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4751
I20260811 02:41:35.215294 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:35.215521 3052 external_mini_cluster.cc:1428] Setting key 9357229c396705198466399489c34031
I20260811 02:41:35.216481 4860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437: Bootstrap starting.
I20260811 02:41:35.217154 4860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:35.217473 4860 log.cc:824] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:35.218309 4860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437: No bootstrap required, opened a new log
I20260811 02:41:35.220654 4860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7934b454d34f36b156178687915437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40259 } }
I20260811 02:41:35.220855 4860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:35.220916 4860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7934b454d34f36b156178687915437, State: Initialized, Role: FOLLOWER
I20260811 02:41:35.221087 4860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7934b454d34f36b156178687915437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40259 } }
I20260811 02:41:35.221194 4860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:35.221235 4860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:35.221278 4860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:35.222041 4860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7934b454d34f36b156178687915437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40259 } }
I20260811 02:41:35.222072 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:35.216297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43345 (local address 127.0.0.1:40259)
0811 02:41:35.216519 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:35.216529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:35.216566 (+ 37us) server_negotiation.cc:407] Connection header received
0811 02:41:35.217080 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:35.217089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:35.217145 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:35.217329 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:35.217656 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:35.218774 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:35.219903 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:35.220361 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:35.220458 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:35.221307 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:35.221318 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:35.221318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:35.221517 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:35.221555 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:35.221568 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:35.221644 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:35.221812 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 02:41:35.221843 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260811 02:41:35.222148 4860 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7934b454d34f36b156178687915437; no voters:
I20260811 02:41:35.222347 4860 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:35.222380 4865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:35.222548 4865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7934b454d34f36b156178687915437, State: Running, Role: LEADER
I20260811 02:41:35.222697 4865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7934b454d34f36b156178687915437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40259 } }
I20260811 02:41:35.222703 4860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:35.223249 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:35.223237 4870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7934b454d34f36b156178687915437. Latest consensus state: current_term: 1 leader_uuid: "7c7934b454d34f36b156178687915437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7934b454d34f36b156178687915437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40259 } } }
I20260811 02:41:35.223326 4870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:35.223709 4866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7934b454d34f36b156178687915437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7934b454d34f36b156178687915437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40259 } } }
I20260811 02:41:35.223912 4875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:35.224182 4866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:35.224588 4875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:35.226495 4875 catalog_manager.cc:1347] Generated new cluster ID: 207de70cc96741e8b91add0f34d2aab6
I20260811 02:41:35.226614 4875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:35.228790 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:35.224929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48389 (local address 127.0.0.1:40259)
0811 02:41:35.224965 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:35.224967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:35.225186 (+ 219us) server_negotiation.cc:407] Connection header received
0811 02:41:35.225245 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:35.225247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:35.225257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:35.225306 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:35.225458 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:35.226201 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:35.227291 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:35.227626 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:35.227711 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:35.228511 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:35.228513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:35.228513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:35.228519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:35.228538 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:35.228543 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:35.228610 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:35.228696 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 02:41:35.228713 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260811 02:41:35.237460 4875 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:35.238147 4875 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:35.243516 4875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7934b454d34f36b156178687915437: Generated new TSK 0
I20260811 02:41:35.243871 4875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:35.247040 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4751
W20260811 02:41:35.263687 4728 connection.cc:535] client connection to 127.0.0.1:40259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40259): Connection reset by peer (error 104)
2026-08-11T02:41:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1559 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:35 dist-test-slave-cvwc krb5kdc[4911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:35 dist-test-slave-cvwc krb5kdc[4911](info): set up 2 sockets
Aug 11 02:41:35 dist-test-slave-cvwc krb5kdc[4911](info): commencing operation
krb5kdc: starting...
W20260811 02:41:36.504930 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:36 dist-test-slave-cvwc krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:36.641435 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:36.729477 4931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:36.729761 4931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:36.729801 4931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:36.731045 4931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:36.731108 4931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:36.731132 4931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:36.731151 4931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:36.731171 4931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:36.732833 4931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44363
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:36.733122 4931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:36.733368 4931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:36.738827 4939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:36.738828 4940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:36.738827 4944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:36.738900 4931 server_base.cc:1034] running on GCE node
I20260811 02:41:36.739331 4931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:36.739617 4931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:36.740797 4931 hybrid_clock.cc:648] HybridClock initialized: now 1786416096740783 us; error 28 us; skew 500 ppm
Aug 11 02:41:36 dist-test-slave-cvwc krb5kdc[4911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:36.743402 4931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:36.743868 4931 webserver.cc:460] Webserver started at http://127.0.0.1:43039/ using document root <none> and password file <none>
I20260811 02:41:36.744122 4931 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:36.744172 4931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:36.744305 4931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:36.745445 4931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "9e006f42ab504500bdd218d658de887c"
format_stamp: "Formatted at 2026-08-11 02:41:36 on dist-test-slave-cvwc"
server_key: "33bd3ef602d21a285da9dd1f9c04fb07"
server_key_iv: "11d8cb7fd629dcd7bba04f33ef35f7df"
server_key_version: "encryptionkey@0"
I20260811 02:41:36.745850 4931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "9e006f42ab504500bdd218d658de887c"
format_stamp: "Formatted at 2026-08-11 02:41:36 on dist-test-slave-cvwc"
server_key: "33bd3ef602d21a285da9dd1f9c04fb07"
server_key_iv: "11d8cb7fd629dcd7bba04f33ef35f7df"
server_key_version: "encryptionkey@0"
I20260811 02:41:36.747370 4931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:36.748610 4952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:36.748890 4931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:36.748986 4931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "9e006f42ab504500bdd218d658de887c"
format_stamp: "Formatted at 2026-08-11 02:41:36 on dist-test-slave-cvwc"
server_key: "33bd3ef602d21a285da9dd1f9c04fb07"
server_key_iv: "11d8cb7fd629dcd7bba04f33ef35f7df"
server_key_version: "encryptionkey@0"
I20260811 02:41:36.749066 4931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:36.752591 4931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:36.754899 4931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:36.755050 4931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:36.759133 4931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44363
I20260811 02:41:36.759254 5035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44363 every 8 connection(s)
I20260811 02:41:36.759542 4931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:36.761104 5036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:36.763952 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c: Bootstrap starting.
I20260811 02:41:36.764621 5036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:36.764906 5036 log.cc:824] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:36.765900 5036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c: No bootstrap required, opened a new log
I20260811 02:41:36.767741 5036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e006f42ab504500bdd218d658de887c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44363 } }
I20260811 02:41:36.767925 5036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:36.767951 5036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e006f42ab504500bdd218d658de887c, State: Initialized, Role: FOLLOWER
I20260811 02:41:36.768121 5036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e006f42ab504500bdd218d658de887c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44363 } }
I20260811 02:41:36.768225 5036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:36.768221 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 4931
I20260811 02:41:36.768265 5036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:36.768329 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:36.768327 5036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:36.768565 3052 external_mini_cluster.cc:1428] Setting key 199714dc28f830027783f735b62ed12d
I20260811 02:41:36.769115 5036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e006f42ab504500bdd218d658de887c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44363 } }
I20260811 02:41:36.769244 5036 leader_election.cc:302] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e006f42ab504500bdd218d658de887c; no voters:
I20260811 02:41:36.769495 5036 leader_election.cc:288] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:36.769531 5040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:36.769840 5040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [term 1 LEADER]: Becoming Leader. State: Replica: 9e006f42ab504500bdd218d658de887c, State: Running, Role: LEADER
I20260811 02:41:36.769906 5036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:36.769955 5040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e006f42ab504500bdd218d658de887c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44363 } }
I20260811 02:41:36.770309 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e006f42ab504500bdd218d658de887c. Latest consensus state: current_term: 1 leader_uuid: "9e006f42ab504500bdd218d658de887c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e006f42ab504500bdd218d658de887c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44363 } } }
I20260811 02:41:36.770399 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:36.770854 5051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:36.770890 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e006f42ab504500bdd218d658de887c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e006f42ab504500bdd218d658de887c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44363 } } }
I20260811 02:41:36.770952 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:36.771399 5051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:36.771853 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:36.769400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51219 (local address 127.0.0.1:44363)
0811 02:41:36.769612 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:36.769624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:36.769673 (+ 49us) server_negotiation.cc:407] Connection header received
0811 02:41:36.769902 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:36.769910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:36.769946 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:36.770464 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:36.771119 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:36.771124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:36.771125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:36.771128 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:36.771168 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:36.771184 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:36.771547 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:36.771585 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 02:41:36.771635 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260811 02:41:36.773092 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:36.773257 5051 catalog_manager.cc:1347] Generated new cluster ID: 29890158a4f94b18b2650cf0e18fb2fe
I20260811 02:41:36.773320 5051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:36.775884 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:36.774721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36433 (local address 127.0.0.1:44363)
0811 02:41:36.774754 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:36.774756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:36.774908 (+ 152us) server_negotiation.cc:407] Connection header received
0811 02:41:36.774960 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:36.774961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:36.774967 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:36.775027 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:36.775660 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:36.775662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:36.775662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:36.775667 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:36.775686 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:36.775692 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:36.775761 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:36.775804 (+ 43us) server_negotiation.cc:299] Negotiation successful
0811 02:41:36.775818 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260811 02:41:36.777794 5051 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:36.777962 5051 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:36.784821 5051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e006f42ab504500bdd218d658de887c: Generated new TSK 0
I20260811 02:41:36.785018 5051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:36.788237 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 4931
2026-08-11T02:41:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1546 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:36 dist-test-slave-cvwc krb5kdc[5089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:36 dist-test-slave-cvwc krb5kdc[5089](info): set up 2 sockets
Aug 11 02:41:36 dist-test-slave-cvwc krb5kdc[5089](info): commencing operation
krb5kdc: starting...
W20260811 02:41:38.035920 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:38 dist-test-slave-cvwc krb5kdc[5089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:38.168453 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:38.255719 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:38.255970 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:38.256033 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:38.257283 5104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:38.257335 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:38.257359 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:38.257378 5104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:38.257398 5104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:38.259230 5104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:38.259527 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:38.259764 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:38.265301 5112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:38.265339 5114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:38.265342 5118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:38.266223 5104 server_base.cc:1034] running on GCE node
I20260811 02:41:38.266512 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:38.266801 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:38.267967 5104 hybrid_clock.cc:648] HybridClock initialized: now 1786416098267948 us; error 37 us; skew 500 ppm
Aug 11 02:41:38 dist-test-slave-cvwc krb5kdc[5089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:38.270749 5104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:38.271291 5104 webserver.cc:460] Webserver started at http://127.0.0.1:38309/ using document root <none> and password file <none>
I20260811 02:41:38.271552 5104 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:38.271608 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:38.271747 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:38.272920 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "f603a89e732541f8b109695fd9047301"
format_stamp: "Formatted at 2026-08-11 02:41:38 on dist-test-slave-cvwc"
server_key: "7b960097c2318aae05b17ca52ba9be67"
server_key_iv: "811059954b631eed975676f25d44d198"
server_key_version: "encryptionkey@0"
I20260811 02:41:38.273317 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "f603a89e732541f8b109695fd9047301"
format_stamp: "Formatted at 2026-08-11 02:41:38 on dist-test-slave-cvwc"
server_key: "7b960097c2318aae05b17ca52ba9be67"
server_key_iv: "811059954b631eed975676f25d44d198"
server_key_version: "encryptionkey@0"
I20260811 02:41:38.274837 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:41:38.275672 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:38.275864 5104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:38.275949 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "f603a89e732541f8b109695fd9047301"
format_stamp: "Formatted at 2026-08-11 02:41:38 on dist-test-slave-cvwc"
server_key: "7b960097c2318aae05b17ca52ba9be67"
server_key_iv: "811059954b631eed975676f25d44d198"
server_key_version: "encryptionkey@0"
I20260811 02:41:38.276022 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:38.291833 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:38.293962 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:38.294106 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:38.298178 5104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40505
I20260811 02:41:38.298301 5222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40505 every 8 connection(s)
I20260811 02:41:38.298758 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:38.299826 5225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:38.302201 5225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301: Bootstrap starting.
I20260811 02:41:38.302903 5225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:38.303154 5225 log.cc:824] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:38.303916 5225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301: No bootstrap required, opened a new log
I20260811 02:41:38.305111 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5104
I20260811 02:41:38.305233 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:38.305403 3052 external_mini_cluster.cc:1428] Setting key 51bc2abde81ba0842f9b568f0183944d
I20260811 02:41:38.305668 5225 raft_consensus.cc:348] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f603a89e732541f8b109695fd9047301" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260811 02:41:38.305889 5225 raft_consensus.cc:374] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:38.305929 5225 raft_consensus.cc:729] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f603a89e732541f8b109695fd9047301, State: Initialized, Role: FOLLOWER
I20260811 02:41:38.306077 5225 consensus_queue.cc:260] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f603a89e732541f8b109695fd9047301" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260811 02:41:38.306155 5225 raft_consensus.cc:388] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:38.306185 5225 raft_consensus.cc:482] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:38.306236 5225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:38.307243 5225 raft_consensus.cc:504] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f603a89e732541f8b109695fd9047301" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260811 02:41:38.307435 5225 leader_election.cc:302] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f603a89e732541f8b109695fd9047301; no voters:
I20260811 02:41:38.307622 5225 leader_election.cc:288] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:38.307655 5233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:38.307761 5233 raft_consensus.cc:686] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [term 1 LEADER]: Becoming Leader. State: Replica: f603a89e732541f8b109695fd9047301, State: Running, Role: LEADER
I20260811 02:41:38.307927 5225 sys_catalog.cc:564] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:38.307901 5233 consensus_queue.cc:237] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f603a89e732541f8b109695fd9047301" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260811 02:41:38.308293 5235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f603a89e732541f8b109695fd9047301. Latest consensus state: current_term: 1 leader_uuid: "f603a89e732541f8b109695fd9047301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f603a89e732541f8b109695fd9047301" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } } }
I20260811 02:41:38.308516 5235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:38.308455 5234 sys_catalog.cc:455] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f603a89e732541f8b109695fd9047301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f603a89e732541f8b109695fd9047301" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } } }
I20260811 02:41:38.308794 5234 sys_catalog.cc:458] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:38.309634 5238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:38.310252 5238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:38.312214 5238 catalog_manager.cc:1347] Generated new cluster ID: 91937368e82a4409b127d8e43aacc4c0
I20260811 02:41:38.312276 5238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:41:38 dist-test-slave-cvwc krb5kdc[5089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 02:41:38.316390 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:38.306014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56567 (local address 127.0.0.1:40505)
0811 02:41:38.306293 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:38.306303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:38.306338 (+ 35us) server_negotiation.cc:407] Connection header received
0811 02:41:38.306454 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:38.306460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:38.306504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:38.306634 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:38.307318 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:38.308613 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:38.309747 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:38.310314 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:38.310434 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:38.313205 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:38.313210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:38.313210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:38.313225 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:38.314878 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:38.315219 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:38.315222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:38.315222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:38.315323 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:38.315511 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:38.315512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:38.315512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:38.315760 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:38.315915 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:38.316014 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 02:41:38.316158 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":123,"threads_started":1}
I20260811 02:41:38.317286 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:38.322606 5238 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:38.323553 5238 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:38.324477 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:38.318630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47791 (local address 127.0.0.1:40505)
0811 02:41:38.318697 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:38.318699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:38.318800 (+ 101us) server_negotiation.cc:407] Connection header received
0811 02:41:38.318859 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:38.318859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:38.318870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:38.318909 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:38.319419 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:38.319903 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:38.320583 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:38.320796 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:38.320860 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:38.321963 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:38.321964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:38.321964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:38.321972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:38.323416 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:38.323796 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:38.323797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:38.323798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:38.323858 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:38.324020 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:38.324020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:38.324021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:38.324128 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:38.324213 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:38.324309 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 02:41:38.324393 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 02:41:38.333276 5238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f603a89e732541f8b109695fd9047301: Generated new TSK 0
I20260811 02:41:38.333485 5238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:38.336040 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5104
2026-08-11T02:41:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:38 dist-test-slave-cvwc krb5kdc[5273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:38 dist-test-slave-cvwc krb5kdc[5273](info): set up 2 sockets
Aug 11 02:41:38 dist-test-slave-cvwc krb5kdc[5273](info): commencing operation
krb5kdc: starting...
W20260811 02:41:39.580355 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:39 dist-test-slave-cvwc krb5kdc[5273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:39.730491 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:39.824922 5292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:39.825261 5292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:39.825469 5292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:39.827118 5292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:39.827183 5292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:39.827198 5292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:39.827210 5292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:39.827221 5292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:39.829216 5292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:39.829483 5292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:39.829703 5292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:39.835405 5304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:39.835729 5300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:39.835609 5301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:39.835548 5292 server_base.cc:1034] running on GCE node
I20260811 02:41:39.836308 5292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:39.836709 5292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:39.837900 5292 hybrid_clock.cc:648] HybridClock initialized: now 1786416099837887 us; error 36 us; skew 500 ppm
Aug 11 02:41:39 dist-test-slave-cvwc krb5kdc[5273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:39.841071 5292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:39.841832 5292 webserver.cc:460] Webserver started at http://127.0.0.1:40541/ using document root <none> and password file <none>
I20260811 02:41:39.842180 5292 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:39.842242 5292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:39.842386 5292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:39.843887 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "6d258249adc749c0a8e44f8049f31767"
format_stamp: "Formatted at 2026-08-11 02:41:39 on dist-test-slave-cvwc"
server_key: "20630e1c441f6cfa690233c84b5a7c24"
server_key_iv: "a6bc9054780cbfa38a6c7f1809ef2911"
server_key_version: "encryptionkey@0"
I20260811 02:41:39.844337 5292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "6d258249adc749c0a8e44f8049f31767"
format_stamp: "Formatted at 2026-08-11 02:41:39 on dist-test-slave-cvwc"
server_key: "20630e1c441f6cfa690233c84b5a7c24"
server_key_iv: "a6bc9054780cbfa38a6c7f1809ef2911"
server_key_version: "encryptionkey@0"
I20260811 02:41:39.846380 5292 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260811 02:41:39.848029 5316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:39.848467 5292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 02:41:39.848618 5292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "6d258249adc749c0a8e44f8049f31767"
format_stamp: "Formatted at 2026-08-11 02:41:39 on dist-test-slave-cvwc"
server_key: "20630e1c441f6cfa690233c84b5a7c24"
server_key_iv: "a6bc9054780cbfa38a6c7f1809ef2911"
server_key_version: "encryptionkey@0"
I20260811 02:41:39.848745 5292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:39.860605 5292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:39.863075 5292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:39.863276 5292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:39.867533 5292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38739
I20260811 02:41:39.867583 5390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38739 every 8 connection(s)
I20260811 02:41:39.867975 5292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:39.869098 5391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:39.871981 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767: Bootstrap starting.
I20260811 02:41:39.872660 5391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:39.872953 5391 log.cc:824] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:39.873863 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767: No bootstrap required, opened a new log
I20260811 02:41:39.875576 5391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d258249adc749c0a8e44f8049f31767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38739 } }
I20260811 02:41:39.875751 5391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:39.875775 5391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d258249adc749c0a8e44f8049f31767, State: Initialized, Role: FOLLOWER
I20260811 02:41:39.875936 5391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d258249adc749c0a8e44f8049f31767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38739 } }
I20260811 02:41:39.876012 5391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:39.876036 5391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:39.876070 5391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:39.876699 5391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d258249adc749c0a8e44f8049f31767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38739 } }
I20260811 02:41:39.876806 5391 leader_election.cc:302] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d258249adc749c0a8e44f8049f31767; no voters:
I20260811 02:41:39.876991 5391 leader_election.cc:288] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:39.877089 5394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:39.877285 5394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [term 1 LEADER]: Becoming Leader. State: Replica: 6d258249adc749c0a8e44f8049f31767, State: Running, Role: LEADER
I20260811 02:41:39.877282 5391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:39.877581 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5292
I20260811 02:41:39.877686 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:39.877871 3052 external_mini_cluster.cc:1428] Setting key 0a4924366e3546d0432819e26170560e
I20260811 02:41:39.877983 5394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d258249adc749c0a8e44f8049f31767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38739 } }
I20260811 02:41:39.878772 5396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d258249adc749c0a8e44f8049f31767. Latest consensus state: current_term: 1 leader_uuid: "6d258249adc749c0a8e44f8049f31767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d258249adc749c0a8e44f8049f31767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38739 } } }
I20260811 02:41:39.878769 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d258249adc749c0a8e44f8049f31767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d258249adc749c0a8e44f8049f31767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38739 } } }
I20260811 02:41:39.879069 5396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:39.879074 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767 [sys.catalog]: This master's current role is: LEADER
W20260811 02:41:39.879374 5422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:41:39.879467 5422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:41:39.879542 5424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:39.879717 5424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:39.881255 5424 catalog_manager.cc:1347] Generated new cluster ID: ea93ab5c195a48fcbcdaedada4b3e994
I20260811 02:41:39.881321 5424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:39.883692 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:39.878385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46835 (local address 127.0.0.1:38739)
0811 02:41:39.878658 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:39.878690 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:39.878798 (+ 108us) server_negotiation.cc:407] Connection header received
0811 02:41:39.879086 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:39.879094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:39.879159 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:39.879323 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:39.879677 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:39.880674 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:39.881809 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:39.882271 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:39.882365 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:39.883042 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:39.883046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:39.883046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:39.883230 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:39.883264 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:39.883277 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:39.883359 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:39.883410 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 02:41:39.883465 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260811 02:41:39.884758 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:39.889971 5424 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:39.889986 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:39.886275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54299 (local address 127.0.0.1:38739)
0811 02:41:39.886318 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:39.886320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:39.886472 (+ 152us) server_negotiation.cc:407] Connection header received
0811 02:41:39.886543 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:39.886544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:39.886556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:39.886611 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:39.886783 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:39.887508 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:39.888510 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:39.888829 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:39.888888 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:39.889668 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:39.889670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:39.889670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:39.889677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:39.889695 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:39.889700 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:39.889802 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:39.889873 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 02:41:39.889905 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:41:39.890664 5424 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:39.896242 5424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d258249adc749c0a8e44f8049f31767: Generated new TSK 0
I20260811 02:41:39.896469 5424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:39.904668 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5292
2026-08-11T02:41:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1592 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:39 dist-test-slave-cvwc krb5kdc[5441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:39 dist-test-slave-cvwc krb5kdc[5441](info): set up 2 sockets
Aug 11 02:41:39 dist-test-slave-cvwc krb5kdc[5441](info): commencing operation
krb5kdc: starting...
W20260811 02:41:41.148078 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:41 dist-test-slave-cvwc krb5kdc[5441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:41.282122 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:41.370148 5460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:41.370402 5460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:41.370440 5460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:41.371644 5460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:41.371701 5460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:41.371733 5460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:41.371752 5460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:41.371773 5460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:41.373347 5460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:41.373624 5460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:41.373886 5460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:41.379269 5467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:41.379428 5460 server_base.cc:1034] running on GCE node
W20260811 02:41:41.379302 5469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:41.379282 5466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:41.379918 5460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:41.380215 5460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:41.381393 5460 hybrid_clock.cc:648] HybridClock initialized: now 1786416101381381 us; error 31 us; skew 500 ppm
Aug 11 02:41:41 dist-test-slave-cvwc krb5kdc[5441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:41.387666 5460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:41.388334 5460 webserver.cc:460] Webserver started at http://127.0.0.1:34883/ using document root <none> and password file <none>
I20260811 02:41:41.388568 5460 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:41.388624 5460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:41.388775 5460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:41.389961 5460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "54055027667a43058450913e59dccaeb"
format_stamp: "Formatted at 2026-08-11 02:41:41 on dist-test-slave-cvwc"
server_key: "d220607b670c2001f7bbdd10c04c5179"
server_key_iv: "a211542a79c2d256ec39e99425cca8a4"
server_key_version: "encryptionkey@0"
I20260811 02:41:41.390367 5460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "54055027667a43058450913e59dccaeb"
format_stamp: "Formatted at 2026-08-11 02:41:41 on dist-test-slave-cvwc"
server_key: "d220607b670c2001f7bbdd10c04c5179"
server_key_iv: "a211542a79c2d256ec39e99425cca8a4"
server_key_version: "encryptionkey@0"
I20260811 02:41:41.391768 5460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:41.392535 5480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:41.392696 5460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260811 02:41:41.392815 5460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "54055027667a43058450913e59dccaeb"
format_stamp: "Formatted at 2026-08-11 02:41:41 on dist-test-slave-cvwc"
server_key: "d220607b670c2001f7bbdd10c04c5179"
server_key_iv: "a211542a79c2d256ec39e99425cca8a4"
server_key_version: "encryptionkey@0"
I20260811 02:41:41.392884 5460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:41.396276 5460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:41.398586 5460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:41.398753 5460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:41.403700 5460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37821
I20260811 02:41:41.403746 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37821 every 8 connection(s)
I20260811 02:41:41.404101 5460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:41.405157 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:41.407550 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb: Bootstrap starting.
I20260811 02:41:41.408082 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5460
I20260811 02:41:41.408215 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:41.408249 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:41.408416 3052 external_mini_cluster.cc:1428] Setting key f80a4a514d260a2bdd91f73aea667b53
I20260811 02:41:41.408565 5567 log.cc:824] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:41.409444 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb: No bootstrap required, opened a new log
I20260811 02:41:41.411284 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:41.408940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40317 (local address 127.0.0.1:37821)
0811 02:41:41.409182 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:41.409192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:41.409230 (+ 38us) server_negotiation.cc:407] Connection header received
0811 02:41:41.409863 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:41.409871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:41.409900 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:41.410028 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:41.410840 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:41.410844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:41.410844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:41.410848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:41.410902 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:41.410913 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:41.410981 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:41.411045 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 02:41:41.411078 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260811 02:41:41.412174 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:41.412135 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54055027667a43058450913e59dccaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37821 } }
I20260811 02:41:41.412319 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:41.412359 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54055027667a43058450913e59dccaeb, State: Initialized, Role: FOLLOWER
I20260811 02:41:41.412467 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54055027667a43058450913e59dccaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37821 } }
I20260811 02:41:41.412547 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:41.412580 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:41.412624 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:41.413362 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54055027667a43058450913e59dccaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37821 } }
I20260811 02:41:41.413480 5567 leader_election.cc:302] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54055027667a43058450913e59dccaeb; no voters:
I20260811 02:41:41.413646 5567 leader_election.cc:288] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:41.413690 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:41.413863 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [term 1 LEADER]: Becoming Leader. State: Replica: 54055027667a43058450913e59dccaeb, State: Running, Role: LEADER
I20260811 02:41:41.414000 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54055027667a43058450913e59dccaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37821 } }
I20260811 02:41:41.413990 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:41.414628 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54055027667a43058450913e59dccaeb. Latest consensus state: current_term: 1 leader_uuid: "54055027667a43058450913e59dccaeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54055027667a43058450913e59dccaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37821 } } }
I20260811 02:41:41.414777 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:41.414944 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:41.413872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33979 (local address 127.0.0.1:37821)
0811 02:41:41.413909 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:41.413911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:41.414011 (+ 100us) server_negotiation.cc:407] Connection header received
0811 02:41:41.414119 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:41.414120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:41.414124 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:41.414170 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:41.414714 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:41.414715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:41.414715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:41.414720 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:41.414735 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:41.414740 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:41.414768 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:41.414876 (+ 108us) server_negotiation.cc:299] Negotiation successful
0811 02:41:41.414889 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:41:41.415030 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54055027667a43058450913e59dccaeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54055027667a43058450913e59dccaeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37821 } } }
I20260811 02:41:41.415189 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:41.415316 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:41.415839 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:41.417418 5594 catalog_manager.cc:1347] Generated new cluster ID: 7f1a348fab9c44e49ea00bf7dadfa555
I20260811 02:41:41.417472 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:41.427188 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:41.427381 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:41.435066 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54055027667a43058450913e59dccaeb: Generated new TSK 0
I20260811 02:41:41.435323 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:41.443920 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5460
2026-08-11T02:41:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:41 dist-test-slave-cvwc krb5kdc[5618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:41 dist-test-slave-cvwc krb5kdc[5618](info): set up 2 sockets
Aug 11 02:41:41 dist-test-slave-cvwc krb5kdc[5618](info): commencing operation
krb5kdc: starting...
W20260811 02:41:42.687327 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:42 dist-test-slave-cvwc krb5kdc[5618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:42.819082 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:42.911862 5642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:42.912097 5642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:42.912124 5642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:42.913345 5642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:42.913386 5642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:42.913400 5642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:42.913411 5642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:42.913424 5642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:42.915050 5642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:42.915266 5642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:42.915457 5642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:42.920686 5655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:42.920686 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:42.920709 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:42.921036 5642 server_base.cc:1034] running on GCE node
I20260811 02:41:42.921300 5642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:42.921619 5642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:42.922783 5642 hybrid_clock.cc:648] HybridClock initialized: now 1786416102922753 us; error 47 us; skew 500 ppm
Aug 11 02:41:42 dist-test-slave-cvwc krb5kdc[5618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:42.925453 5642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:42.925961 5642 webserver.cc:460] Webserver started at http://127.0.0.1:36491/ using document root <none> and password file <none>
I20260811 02:41:42.926448 5642 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:42.926499 5642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:42.926621 5642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:42.927713 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "c64ffd577b99404aad411aae3aac59b5"
format_stamp: "Formatted at 2026-08-11 02:41:42 on dist-test-slave-cvwc"
server_key: "41a9051962126834b51470f05105b84d"
server_key_iv: "3923d73d0ae6a79d21ed7dc7a5c446fd"
server_key_version: "encryptionkey@0"
I20260811 02:41:42.928203 5642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "c64ffd577b99404aad411aae3aac59b5"
format_stamp: "Formatted at 2026-08-11 02:41:42 on dist-test-slave-cvwc"
server_key: "41a9051962126834b51470f05105b84d"
server_key_iv: "3923d73d0ae6a79d21ed7dc7a5c446fd"
server_key_version: "encryptionkey@0"
I20260811 02:41:42.929592 5642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:42.930454 5667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:42.930619 5642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:42.930720 5642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "c64ffd577b99404aad411aae3aac59b5"
format_stamp: "Formatted at 2026-08-11 02:41:42 on dist-test-slave-cvwc"
server_key: "41a9051962126834b51470f05105b84d"
server_key_iv: "3923d73d0ae6a79d21ed7dc7a5c446fd"
server_key_version: "encryptionkey@0"
I20260811 02:41:42.930789 5642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:42.942406 5642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:42.944715 5642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:42.944844 5642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:42.949451 5642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34847
I20260811 02:41:42.949527 5750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34847 every 8 connection(s)
I20260811 02:41:42.949947 5642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:42.951041 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:42.953438 5754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5: Bootstrap starting.
I20260811 02:41:42.954118 5754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:42.954399 5754 log.cc:824] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:42.955210 5754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5: No bootstrap required, opened a new log
I20260811 02:41:42.955492 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5642
I20260811 02:41:42.955593 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:42.955754 3052 external_mini_cluster.cc:1428] Setting key 6b832f334838421e9f3e5ada7b2f9267
I20260811 02:41:42.957329 5754 raft_consensus.cc:348] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64ffd577b99404aad411aae3aac59b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260811 02:41:42.957521 5754 raft_consensus.cc:374] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:42.957557 5754 raft_consensus.cc:729] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64ffd577b99404aad411aae3aac59b5, State: Initialized, Role: FOLLOWER
I20260811 02:41:42.957687 5754 consensus_queue.cc:260] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64ffd577b99404aad411aae3aac59b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260811 02:41:42.957795 5754 raft_consensus.cc:388] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:42.957823 5754 raft_consensus.cc:482] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:42.957871 5754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:42.959019 5754 raft_consensus.cc:504] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64ffd577b99404aad411aae3aac59b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260811 02:41:42.959136 5754 leader_election.cc:302] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c64ffd577b99404aad411aae3aac59b5; no voters:
I20260811 02:41:42.959296 5754 leader_election.cc:288] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:42.959343 5765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:42.959546 5754 sys_catalog.cc:564] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:42.959640 5765 raft_consensus.cc:686] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [term 1 LEADER]: Becoming Leader. State: Replica: c64ffd577b99404aad411aae3aac59b5, State: Running, Role: LEADER
I20260811 02:41:42.959741 5765 consensus_queue.cc:237] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64ffd577b99404aad411aae3aac59b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260811 02:41:42.960086 5765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c64ffd577b99404aad411aae3aac59b5. Latest consensus state: current_term: 1 leader_uuid: "c64ffd577b99404aad411aae3aac59b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64ffd577b99404aad411aae3aac59b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } } }
I20260811 02:41:42.960167 5765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:42.960367 5766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c64ffd577b99404aad411aae3aac59b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c64ffd577b99404aad411aae3aac59b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } } }
I20260811 02:41:42.960448 5766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:42.960378 5781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:42.961135 5781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:42.964385 5781 catalog_manager.cc:1347] Generated new cluster ID: 3c1ca95790d74d0d95c23d8b49397264
I20260811 02:41:42.964437 5781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:41:42 dist-test-slave-cvwc krb5kdc[5618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 02:41:42.969605 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:42.956423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:54557 (local address 127.0.0.1:34847)
0811 02:41:42.956674 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:42.956685 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:42.956722 (+ 37us) server_negotiation.cc:407] Connection header received
0811 02:41:42.957282 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:42.957291 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:42.957363 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:42.957530 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:42.958274 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:42.959667 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:42.960804 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:42.961920 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:42.962039 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:42.965250 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:42.965255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:42.965255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:42.965267 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:42.967488 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:42.968119 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:42.968124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:42.968124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:42.968417 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:42.968677 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:42.968678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:42.968678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:42.968999 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:42.969151 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:42.969284 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 02:41:42.969392 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":119,"threads_started":1}
I20260811 02:41:42.970657 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:42.974334 5781 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:42.975068 5781 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:42.978392 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:42.972129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:36411 (local address 127.0.0.1:34847)
0811 02:41:42.972166 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:42.972168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:42.972222 (+ 54us) server_negotiation.cc:407] Connection header received
0811 02:41:42.972288 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:42.972289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:42.972300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:42.972348 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:42.972982 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:42.973809 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:42.974611 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:42.974833 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:42.974889 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:42.976049 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:42.976050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:42.976051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:42.976058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:42.977273 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:42.977602 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:42.977603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:42.977603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:42.977651 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:42.977868 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:42.977869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:42.977869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:42.977967 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:42.978060 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:42.978211 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 02:41:42.978288 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":19}
I20260811 02:41:42.978569 5781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c64ffd577b99404aad411aae3aac59b5: Generated new TSK 0
I20260811 02:41:42.978744 5781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:42.982434 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5642
W20260811 02:41:43.003415 5613 connection.cc:535] client connection to 127.0.0.1:34847 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34847): Connection reset by peer (error 104)
2026-08-11T02:41:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1540 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:43 dist-test-slave-cvwc krb5kdc[5810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:43 dist-test-slave-cvwc krb5kdc[5810](info): set up 2 sockets
Aug 11 02:41:43 dist-test-slave-cvwc krb5kdc[5810](info): commencing operation
krb5kdc: starting...
W20260811 02:41:44.206790 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:44 dist-test-slave-cvwc krb5kdc[5810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:44.341650 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:44.428296 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:44.428505 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:44.428537 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:44.429740 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:44.429785 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:44.429813 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:44.429837 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:44.429862 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:44.431365 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:44.431603 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:44.431813 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:44.436544 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:44.436600 5839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:44.436615 5838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:44.436695 5830 server_base.cc:1034] running on GCE node
I20260811 02:41:44.437096 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:44.437410 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:44.438566 5830 hybrid_clock.cc:648] HybridClock initialized: now 1786416104438552 us; error 28 us; skew 500 ppm
Aug 11 02:41:44 dist-test-slave-cvwc krb5kdc[5810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:44.441071 5830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:44.441507 5830 webserver.cc:460] Webserver started at http://127.0.0.1:35261/ using document root <none> and password file <none>
I20260811 02:41:44.441776 5830 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:44.441824 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:44.441943 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:44.443008 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7"
format_stamp: "Formatted at 2026-08-11 02:41:44 on dist-test-slave-cvwc"
server_key: "0d7b3d7f4bbd3ad52a136e92ce75ffa0"
server_key_iv: "47991e199da09ba8da887c5fba3c0d76"
server_key_version: "encryptionkey@0"
I20260811 02:41:44.443337 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7"
format_stamp: "Formatted at 2026-08-11 02:41:44 on dist-test-slave-cvwc"
server_key: "0d7b3d7f4bbd3ad52a136e92ce75ffa0"
server_key_iv: "47991e199da09ba8da887c5fba3c0d76"
server_key_version: "encryptionkey@0"
I20260811 02:41:44.444628 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260811 02:41:44.445430 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:44.445616 5830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:44.445695 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7"
format_stamp: "Formatted at 2026-08-11 02:41:44 on dist-test-slave-cvwc"
server_key: "0d7b3d7f4bbd3ad52a136e92ce75ffa0"
server_key_iv: "47991e199da09ba8da887c5fba3c0d76"
server_key_version: "encryptionkey@0"
I20260811 02:41:44.445796 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:44.456694 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:44.458842 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:44.458990 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:44.463395 5830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41797
I20260811 02:41:44.463445 5940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41797 every 8 connection(s)
I20260811 02:41:44.463788 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:44.464814 5942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:44.467226 5942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7: Bootstrap starting.
I20260811 02:41:44.467504 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 5830
I20260811 02:41:44.467608 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:44.467834 3052 external_mini_cluster.cc:1428] Setting key 27511755619710ff003944b8e45fd58a
I20260811 02:41:44.467862 5942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:44.468171 5942 log.cc:824] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:44.469027 5942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7: No bootstrap required, opened a new log
I20260811 02:41:44.471647 5942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41797 } }
I20260811 02:41:44.471855 5942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:44.471891 5942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd2ff8b3a8a4adbb1f0a4149717c0e7, State: Initialized, Role: FOLLOWER
I20260811 02:41:44.472039 5942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41797 } }
I20260811 02:41:44.472122 5942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:44.472155 5942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:44.472206 5942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:44.472872 5942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41797 } }
I20260811 02:41:44.472986 5942 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd2ff8b3a8a4adbb1f0a4149717c0e7; no voters:
I20260811 02:41:44.473176 5942 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:44.473263 5948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:44.473464 5942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:44.473479 5948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd2ff8b3a8a4adbb1f0a4149717c0e7, State: Running, Role: LEADER
I20260811 02:41:44.473801 5948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41797 } }
I20260811 02:41:44.474208 5950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd2ff8b3a8a4adbb1f0a4149717c0e7. Latest consensus state: current_term: 1 leader_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41797 } } }
I20260811 02:41:44.474577 5950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:44.474960 5961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:44.475025 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd2ff8b3a8a4adbb1f0a4149717c0e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41797 } } }
I20260811 02:41:44.475243 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:44.475600 5961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:44.475538 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:44.468438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:58969 (local address 127.0.0.1:41797)
0811 02:41:44.468627 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:44.468636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:44.468807 (+ 171us) server_negotiation.cc:407] Connection header received
0811 02:41:44.469410 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:44.469420 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:44.469490 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:44.469818 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:44.470511 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:44.471882 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:44.472749 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:44.473384 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:44.473496 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:44.474593 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:44.474598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:44.474598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:44.474820 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:44.474863 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:44.474882 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:44.474979 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:44.475282 (+ 303us) server_negotiation.cc:299] Negotiation successful
0811 02:41:44.475314 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260811 02:41:44.476758 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:44.477468 5961 catalog_manager.cc:1347] Generated new cluster ID: a326175f6445467d88d14ac261d91c67
I20260811 02:41:44.477524 5961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:44.481582 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:44.478375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:40851 (local address 127.0.0.1:41797)
0811 02:41:44.478414 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:44.478416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:44.478538 (+ 122us) server_negotiation.cc:407] Connection header received
0811 02:41:44.478587 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:44.478588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:44.478599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:44.478662 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:44.478774 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:44.479514 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:44.480266 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:44.480566 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:44.480619 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:44.481273 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:44.481274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:44.481275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:44.481279 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:44.481290 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:44.481294 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:44.481339 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:44.481488 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 02:41:44.481498 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:41:44.507172 5961 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:44.507756 5961 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:44.510667 5961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd2ff8b3a8a4adbb1f0a4149717c0e7: Generated new TSK 0
I20260811 02:41:44.510843 5961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:44.519734 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 5830
W20260811 02:41:44.544932 5808 connection.cc:535] client connection to 127.0.0.1:41797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41797): Connection reset by peer (error 104)
2026-08-11T02:41:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:44 dist-test-slave-cvwc krb5kdc[5986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:44 dist-test-slave-cvwc krb5kdc[5986](info): set up 2 sockets
Aug 11 02:41:44 dist-test-slave-cvwc krb5kdc[5986](info): commencing operation
krb5kdc: starting...
W20260811 02:41:45.752328 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:45 dist-test-slave-cvwc krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:45.885300 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:45.972436 6003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:45.972669 6003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:45.972694 6003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:45.973927 6003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:45.973985 6003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:45.973999 6003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:45.974010 6003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:45.974021 6003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:45.975592 6003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:45.975863 6003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:45.976114 6003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:45.981336 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:45.981309 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:45.981313 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:45.981472 6003 server_base.cc:1034] running on GCE node
I20260811 02:41:45.981884 6003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:45.982182 6003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:45.983352 6003 hybrid_clock.cc:648] HybridClock initialized: now 1786416105983330 us; error 43 us; skew 500 ppm
Aug 11 02:41:45 dist-test-slave-cvwc krb5kdc[5986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:45.985970 6003 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:45.986436 6003 webserver.cc:460] Webserver started at http://127.0.0.1:36409/ using document root <none> and password file <none>
I20260811 02:41:45.986658 6003 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:45.986711 6003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:45.986846 6003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:45.987958 6003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "416293945c8d4ea0af0a54fde37ba0c5"
format_stamp: "Formatted at 2026-08-11 02:41:45 on dist-test-slave-cvwc"
server_key: "a50337eb0f8796456b64f122a69bff40"
server_key_iv: "46465810889b5b4ce294fb507830b329"
server_key_version: "encryptionkey@0"
I20260811 02:41:45.988332 6003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "416293945c8d4ea0af0a54fde37ba0c5"
format_stamp: "Formatted at 2026-08-11 02:41:45 on dist-test-slave-cvwc"
server_key: "a50337eb0f8796456b64f122a69bff40"
server_key_iv: "46465810889b5b4ce294fb507830b329"
server_key_version: "encryptionkey@0"
I20260811 02:41:45.989797 6003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260811 02:41:45.990804 6025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:45.991022 6003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:45.991117 6003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "416293945c8d4ea0af0a54fde37ba0c5"
format_stamp: "Formatted at 2026-08-11 02:41:45 on dist-test-slave-cvwc"
server_key: "a50337eb0f8796456b64f122a69bff40"
server_key_iv: "46465810889b5b4ce294fb507830b329"
server_key_version: "encryptionkey@0"
I20260811 02:41:45.991199 6003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:45.994721 6003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:45.996898 6003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:45.997035 6003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:46.000952 6003 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34347
I20260811 02:41:46.000998 6117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34347 every 8 connection(s)
I20260811 02:41:46.001652 6003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:46.002108 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6003
I20260811 02:41:46.002211 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:46.002359 3052 external_mini_cluster.cc:1428] Setting key 8f291dc125adbc6f414edb088cb1d56a
I20260811 02:41:46.003443 6118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:46.005497 6122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:46.003184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:55243 (local address 127.0.0.1:34347)
0811 02:41:46.003370 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:46.003380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:46.003406 (+ 26us) server_negotiation.cc:407] Connection header received
0811 02:41:46.003953 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:46.003961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:46.003989 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:46.004106 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:46.005020 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:46.005024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:46.005024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:46.005028 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:46.005072 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:46.005084 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:46.005156 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:46.005235 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 02:41:46.005292 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260811 02:41:46.006449 6118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5: Bootstrap starting.
I20260811 02:41:46.006944 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:46.007146 6118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:46.007476 6118 log.cc:824] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:46.008353 6118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5: No bootstrap required, opened a new log
I20260811 02:41:46.010284 6122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:46.008874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:53605 (local address 127.0.0.1:34347)
0811 02:41:46.008968 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:46.008971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:46.009086 (+ 115us) server_negotiation.cc:407] Connection header received
0811 02:41:46.009162 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:46.009163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:46.009169 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:46.009223 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:46.010076 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:46.010078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:46.010078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:46.010082 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:46.010097 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:46.010101 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:46.010156 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:46.010212 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 02:41:46.010224 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 02:41:46.010711 6118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416293945c8d4ea0af0a54fde37ba0c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34347 } }
I20260811 02:41:46.010859 6118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:46.010890 6118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416293945c8d4ea0af0a54fde37ba0c5, State: Initialized, Role: FOLLOWER
I20260811 02:41:46.011019 6118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416293945c8d4ea0af0a54fde37ba0c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34347 } }
I20260811 02:41:46.011094 6118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:46.011125 6118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:46.011175 6118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:46.011758 6118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416293945c8d4ea0af0a54fde37ba0c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34347 } }
I20260811 02:41:46.011862 6118 leader_election.cc:302] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416293945c8d4ea0af0a54fde37ba0c5; no voters:
I20260811 02:41:46.012050 6118 leader_election.cc:288] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:46.012100 6132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:46.012264 6132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [term 1 LEADER]: Becoming Leader. State: Replica: 416293945c8d4ea0af0a54fde37ba0c5, State: Running, Role: LEADER
I20260811 02:41:46.012318 6118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:46.012357 6132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416293945c8d4ea0af0a54fde37ba0c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34347 } }
I20260811 02:41:46.012689 6133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416293945c8d4ea0af0a54fde37ba0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416293945c8d4ea0af0a54fde37ba0c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34347 } } }
I20260811 02:41:46.012789 6133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:46.012953 6134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416293945c8d4ea0af0a54fde37ba0c5. Latest consensus state: current_term: 1 leader_uuid: "416293945c8d4ea0af0a54fde37ba0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416293945c8d4ea0af0a54fde37ba0c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34347 } } }
I20260811 02:41:46.013015 6134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:46.012998 6142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:46.013798 6142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:46.015300 6142 catalog_manager.cc:1347] Generated new cluster ID: 326feb44c15e4001ad7889d98619c53f
I20260811 02:41:46.015352 6142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:46.020488 6142 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:46.020658 6142 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:46.030125 6142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416293945c8d4ea0af0a54fde37ba0c5: Generated new TSK 0
I20260811 02:41:46.030369 6142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:46.036157 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6003
2026-08-11T02:41:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:46 dist-test-slave-cvwc krb5kdc[6171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:46 dist-test-slave-cvwc krb5kdc[6171](info): set up 2 sockets
Aug 11 02:41:46 dist-test-slave-cvwc krb5kdc[6171](info): commencing operation
krb5kdc: starting...
W20260811 02:41:47.257123 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:47 dist-test-slave-cvwc krb5kdc[6171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:47.389858 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:47.481400 6191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:47.481652 6191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:47.481676 6191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:47.483173 6191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:47.483255 6191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:47.483271 6191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:47.483283 6191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:47.483294 6191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:47.484998 6191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:47.485289 6191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:47.485526 6191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:47.490882 6198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:47.490851 6199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:47.491153 6191 server_base.cc:1034] running on GCE node
W20260811 02:41:47.490851 6203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:47.491518 6191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:47.491852 6191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:47.493028 6191 hybrid_clock.cc:648] HybridClock initialized: now 1786416107493013 us; error 33 us; skew 500 ppm
Aug 11 02:41:47 dist-test-slave-cvwc krb5kdc[6171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:47.500586 6191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:47.501088 6191 webserver.cc:460] Webserver started at http://127.0.0.1:36669/ using document root <none> and password file <none>
I20260811 02:41:47.501320 6191 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:47.501374 6191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:47.501509 6191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:47.503086 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "9094c3cf34a84166898aa429095312a9"
format_stamp: "Formatted at 2026-08-11 02:41:47 on dist-test-slave-cvwc"
server_key: "d6a16b1be3022f11b7344e83d3fa5c78"
server_key_iv: "c3528452ddeac8357a08f677c5c0531f"
server_key_version: "encryptionkey@0"
I20260811 02:41:47.503520 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "9094c3cf34a84166898aa429095312a9"
format_stamp: "Formatted at 2026-08-11 02:41:47 on dist-test-slave-cvwc"
server_key: "d6a16b1be3022f11b7344e83d3fa5c78"
server_key_iv: "c3528452ddeac8357a08f677c5c0531f"
server_key_version: "encryptionkey@0"
I20260811 02:41:47.505143 6191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:47.506245 6216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:47.506561 6191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:47.506672 6191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "9094c3cf34a84166898aa429095312a9"
format_stamp: "Formatted at 2026-08-11 02:41:47 on dist-test-slave-cvwc"
server_key: "d6a16b1be3022f11b7344e83d3fa5c78"
server_key_iv: "c3528452ddeac8357a08f677c5c0531f"
server_key_version: "encryptionkey@0"
I20260811 02:41:47.506754 6191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:47.522598 6191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:47.525441 6191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:47.525580 6191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:47.529589 6191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42091
I20260811 02:41:47.529637 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42091 every 8 connection(s)
I20260811 02:41:47.530030 6191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:47.531150 6300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:47.533669 6300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9: Bootstrap starting.
I20260811 02:41:47.534471 6300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:47.534724 6300 log.cc:824] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:47.535573 6300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9: No bootstrap required, opened a new log
I20260811 02:41:47.536103 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6191
I20260811 02:41:47.536219 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:47.536412 3052 external_mini_cluster.cc:1428] Setting key fc8b4131c928053b9d1e64a9f9d07652
I20260811 02:41:47.538036 6300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9094c3cf34a84166898aa429095312a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260811 02:41:47.538203 6300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:47.538224 6300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9094c3cf34a84166898aa429095312a9, State: Initialized, Role: FOLLOWER
I20260811 02:41:47.538342 6300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9094c3cf34a84166898aa429095312a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260811 02:41:47.538427 6300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:47.538518 6300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:47.538581 6300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:47.539361 6300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9094c3cf34a84166898aa429095312a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260811 02:41:47.539459 6300 leader_election.cc:302] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9094c3cf34a84166898aa429095312a9; no voters:
I20260811 02:41:47.539633 6300 leader_election.cc:288] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:47.539678 6308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:47.539834 6308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [term 1 LEADER]: Becoming Leader. State: Replica: 9094c3cf34a84166898aa429095312a9, State: Running, Role: LEADER
I20260811 02:41:47.539961 6300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:47.539973 6308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9094c3cf34a84166898aa429095312a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } }
I20260811 02:41:47.540582 6309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9094c3cf34a84166898aa429095312a9. Latest consensus state: current_term: 1 leader_uuid: "9094c3cf34a84166898aa429095312a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9094c3cf34a84166898aa429095312a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } } }
I20260811 02:41:47.540671 6309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:47.540901 6310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9094c3cf34a84166898aa429095312a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9094c3cf34a84166898aa429095312a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42091 } } }
I20260811 02:41:47.540983 6310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9 [sys.catalog]: This master's current role is: LEADER
W20260811 02:41:47.541868 6334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:41:47.541927 6334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:41:47.541967 6323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:47.542130 6323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:47.543780 6323 catalog_manager.cc:1347] Generated new cluster ID: af78e07104304525a1f619a27ea776ff
I20260811 02:41:47.543838 6323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:41:47 dist-test-slave-cvwc krb5kdc[6171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 02:41:47.549880 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:47.537176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:39227 (local address 127.0.0.1:42091)
0811 02:41:47.537369 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:47.537377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:47.537406 (+ 29us) server_negotiation.cc:407] Connection header received
0811 02:41:47.538184 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:47.538193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:47.538282 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:47.538431 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:47.539245 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:47.540638 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:47.541760 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:47.542264 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:47.542377 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:47.545148 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:47.545153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:47.545154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:47.545168 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:47.547593 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:47.548151 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:47.548155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:47.548155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:47.548278 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:47.548521 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:47.548522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:47.548522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:47.548970 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:47.549242 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:47.549487 (+ 245us) server_negotiation.cc:299] Negotiation successful
0811 02:41:47.549624 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260811 02:41:47.551021 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:47.559907 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:47.552688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:42125 (local address 127.0.0.1:42091)
0811 02:41:47.552734 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:47.552736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:47.552846 (+ 110us) server_negotiation.cc:407] Connection header received
0811 02:41:47.552918 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:47.552919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:47.552929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:47.552978 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:47.553539 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:47.554131 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:47.554856 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:47.555102 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:47.555162 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:47.556416 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:47.556417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:47.556418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:47.556426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:47.558239 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:47.558745 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:47.558746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:47.558747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:47.558812 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:47.559043 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:47.559044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:47.559044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:47.559289 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:47.559460 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:47.559676 (+ 216us) server_negotiation.cc:299] Negotiation successful
0811 02:41:47.559775 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:41:47.563180 6323 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:47.563760 6323 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:47.569078 6323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9094c3cf34a84166898aa429095312a9: Generated new TSK 0
I20260811 02:41:47.569366 6323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:47.576721 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6191
W20260811 02:41:47.602008 6169 connection.cc:535] client connection to 127.0.0.1:42091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42091): Connection reset by peer (error 104)
2026-08-11T02:41:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1546 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:47 dist-test-slave-cvwc krb5kdc[6354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:47 dist-test-slave-cvwc krb5kdc[6354](info): set up 2 sockets
Aug 11 02:41:47 dist-test-slave-cvwc krb5kdc[6354](info): commencing operation
krb5kdc: starting...
W20260811 02:41:48.819871 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:48 dist-test-slave-cvwc krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:48.953317 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:49.037580 6371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:49.037842 6371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:49.037876 6371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:49.039103 6371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:49.039157 6371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:49.039177 6371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:49.039193 6371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:49.039209 6371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:49.040879 6371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:49.041146 6371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:49.041370 6371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:49.046491 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:49.046505 6384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:49.046636 6371 server_base.cc:1034] running on GCE node
W20260811 02:41:49.046505 6382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:49.046970 6371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:49.047287 6371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:49.048475 6371 hybrid_clock.cc:648] HybridClock initialized: now 1786416109048443 us; error 46 us; skew 500 ppm
Aug 11 02:41:49 dist-test-slave-cvwc krb5kdc[6354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:49.051265 6371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:49.051769 6371 webserver.cc:460] Webserver started at http://127.0.0.1:42021/ using document root <none> and password file <none>
I20260811 02:41:49.052009 6371 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:49.052052 6371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:49.052147 6371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:49.053225 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "d64963b461074410be2ce91f71e59b41"
format_stamp: "Formatted at 2026-08-11 02:41:49 on dist-test-slave-cvwc"
server_key: "688e1a36b935390a0096128e4cc0f034"
server_key_iv: "c5ee606a45aa513a6c38a1b29dfc0184"
server_key_version: "encryptionkey@0"
I20260811 02:41:49.053548 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "d64963b461074410be2ce91f71e59b41"
format_stamp: "Formatted at 2026-08-11 02:41:49 on dist-test-slave-cvwc"
server_key: "688e1a36b935390a0096128e4cc0f034"
server_key_iv: "c5ee606a45aa513a6c38a1b29dfc0184"
server_key_version: "encryptionkey@0"
I20260811 02:41:49.054965 6371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:41:49.055821 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:49.056102 6371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:49.056211 6371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "d64963b461074410be2ce91f71e59b41"
format_stamp: "Formatted at 2026-08-11 02:41:49 on dist-test-slave-cvwc"
server_key: "688e1a36b935390a0096128e4cc0f034"
server_key_iv: "c5ee606a45aa513a6c38a1b29dfc0184"
server_key_version: "encryptionkey@0"
I20260811 02:41:49.056303 6371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:49.077832 6371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:49.080176 6371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:49.080332 6371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:49.084676 6371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43431
I20260811 02:41:49.084709 6488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43431 every 8 connection(s)
I20260811 02:41:49.085101 6371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:49.086022 6489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:49.088454 6489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41: Bootstrap starting.
I20260811 02:41:49.089231 6489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:49.089560 6489 log.cc:824] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:49.089797 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6371
I20260811 02:41:49.089922 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:49.090108 3052 external_mini_cluster.cc:1428] Setting key 42a4301c931f13202abc38a466eada1e
I20260811 02:41:49.090363 6489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41: No bootstrap required, opened a new log
I20260811 02:41:49.093200 6489 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64963b461074410be2ce91f71e59b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260811 02:41:49.093400 6489 raft_consensus.cc:374] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:49.093433 6489 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64963b461074410be2ce91f71e59b41, State: Initialized, Role: FOLLOWER
I20260811 02:41:49.093564 6489 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64963b461074410be2ce91f71e59b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260811 02:41:49.093650 6489 raft_consensus.cc:388] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:49.093676 6489 raft_consensus.cc:482] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:49.093750 6489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:49.094372 6489 raft_consensus.cc:504] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64963b461074410be2ce91f71e59b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260811 02:41:49.094484 6489 leader_election.cc:302] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d64963b461074410be2ce91f71e59b41; no voters:
I20260811 02:41:49.094663 6489 leader_election.cc:288] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:49.094767 6499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:49.094933 6489 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:49.095037 6499 raft_consensus.cc:686] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [term 1 LEADER]: Becoming Leader. State: Replica: d64963b461074410be2ce91f71e59b41, State: Running, Role: LEADER
I20260811 02:41:49.095177 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64963b461074410be2ce91f71e59b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260811 02:41:49.095603 6500 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d64963b461074410be2ce91f71e59b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64963b461074410be2ce91f71e59b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } } }
I20260811 02:41:49.095598 6501 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d64963b461074410be2ce91f71e59b41. Latest consensus state: current_term: 1 leader_uuid: "d64963b461074410be2ce91f71e59b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64963b461074410be2ce91f71e59b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } } }
I20260811 02:41:49.095706 6500 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:49.095718 6501 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:49.096040 6516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:49.096808 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:49.090624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:57127 (local address 127.0.0.1:43431)
0811 02:41:49.090861 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:49.090872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:49.090907 (+ 35us) server_negotiation.cc:407] Connection header received
0811 02:41:49.091699 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:49.091710 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:49.091779 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:49.091945 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:49.092115 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:49.093473 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:49.094397 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:49.095023 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:49.095487 (+ 464us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:49.095902 (+ 415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:49.095907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:49.095908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:49.096128 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:49.096168 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:49.096184 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:49.096510 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:49.096557 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 02:41:49.096588 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":114,"threads_started":1}
I20260811 02:41:49.096814 6516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:49.098254 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:49.098742 6516 catalog_manager.cc:1347] Generated new cluster ID: 4673ff8e41cd46779637937e1d395743
I20260811 02:41:49.098807 6516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:49.107599 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:49.102070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:35017 (local address 127.0.0.1:43431)
0811 02:41:49.102106 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:49.102108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:49.102224 (+ 116us) server_negotiation.cc:407] Connection header received
0811 02:41:49.102318 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:49.102319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:49.102328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:49.102354 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:49.102539 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:49.104155 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:49.105491 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:49.105896 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:49.105943 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:49.107296 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:49.107299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:49.107299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:49.107307 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:49.107330 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:49.107336 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:49.107371 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:49.107489 (+ 118us) server_negotiation.cc:299] Negotiation successful
0811 02:41:49.107504 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260811 02:41:49.110168 6516 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:49.110976 6516 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:49.131084 6516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d64963b461074410be2ce91f71e59b41: Generated new TSK 0
I20260811 02:41:49.131326 6516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:49.142247 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6371
W20260811 02:41:49.165756 6350 connection.cc:535] client connection to 127.0.0.1:43431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43431): Connection reset by peer (error 104)
2026-08-11T02:41:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1563 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:49 dist-test-slave-cvwc krb5kdc[6558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:49 dist-test-slave-cvwc krb5kdc[6558](info): set up 2 sockets
Aug 11 02:41:49 dist-test-slave-cvwc krb5kdc[6558](info): commencing operation
krb5kdc: starting...
W20260811 02:41:50.344512 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:50 dist-test-slave-cvwc krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 02:41:50.481462 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:50.571303 6580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:50.571573 6580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:50.571609 6580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:50.572819 6580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:50.572868 6580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:50.572888 6580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:50.572904 6580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:50.572921 6580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:50.574540 6580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:50.574819 6580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:50.575044 6580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:50.580233 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:50.580227 6592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:50.580227 6596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:50.580252 6580 server_base.cc:1034] running on GCE node
I20260811 02:41:50.580749 6580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:50.581067 6580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:50.582239 6580 hybrid_clock.cc:648] HybridClock initialized: now 1786416110582223 us; error 30 us; skew 500 ppm
Aug 11 02:41:50 dist-test-slave-cvwc krb5kdc[6558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:50.585088 6580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 02:41:50.585662 6580 webserver.cc:460] Webserver started at http://127.0.0.1:41461/ using document root <none> and password file <none>
I20260811 02:41:50.585970 6580 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:50.586025 6580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:50.586136 6580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:50.587246 6580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "7f76c7efa3e440dcac16d4ccaf0f055e"
format_stamp: "Formatted at 2026-08-11 02:41:50 on dist-test-slave-cvwc"
server_key: "f6be44370d48d6f97a001379fd217c25"
server_key_iv: "d43aaffe910d6bd10cfd47410fa254e5"
server_key_version: "encryptionkey@0"
I20260811 02:41:50.588064 6580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "7f76c7efa3e440dcac16d4ccaf0f055e"
format_stamp: "Formatted at 2026-08-11 02:41:50 on dist-test-slave-cvwc"
server_key: "f6be44370d48d6f97a001379fd217c25"
server_key_iv: "d43aaffe910d6bd10cfd47410fa254e5"
server_key_version: "encryptionkey@0"
I20260811 02:41:50.590083 6580 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260811 02:41:50.591464 6606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:50.591699 6580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:50.591807 6580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "7f76c7efa3e440dcac16d4ccaf0f055e"
format_stamp: "Formatted at 2026-08-11 02:41:50 on dist-test-slave-cvwc"
server_key: "f6be44370d48d6f97a001379fd217c25"
server_key_iv: "d43aaffe910d6bd10cfd47410fa254e5"
server_key_version: "encryptionkey@0"
I20260811 02:41:50.591893 6580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:50.595619 6580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:50.597950 6580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:50.598105 6580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:50.602512 6580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42051
I20260811 02:41:50.602530 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42051 every 8 connection(s)
I20260811 02:41:50.602967 6580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:50.603936 6693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:50.606513 6693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e: Bootstrap starting.
I20260811 02:41:50.607092 6693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:50.607331 6693 log.cc:824] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:50.608119 6693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e: No bootstrap required, opened a new log
I20260811 02:41:50.608660 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6580
I20260811 02:41:50.608796 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:50.608975 3052 external_mini_cluster.cc:1428] Setting key dc946e1d2762fcd3502a3953d70b560f
I20260811 02:41:50.610288 6693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260811 02:41:50.610460 6693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:50.610483 6693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f76c7efa3e440dcac16d4ccaf0f055e, State: Initialized, Role: FOLLOWER
I20260811 02:41:50.610616 6693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260811 02:41:50.610697 6693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:50.610731 6693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:50.610828 6693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:50.611683 6693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260811 02:41:50.611811 6693 leader_election.cc:302] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f76c7efa3e440dcac16d4ccaf0f055e; no voters:
I20260811 02:41:50.611997 6693 leader_election.cc:288] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:50.612032 6699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:50.612273 6699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [term 1 LEADER]: Becoming Leader. State: Replica: 7f76c7efa3e440dcac16d4ccaf0f055e, State: Running, Role: LEADER
I20260811 02:41:50.612393 6693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:50.612421 6699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260811 02:41:50.612468 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:50.609765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:55947 (local address 127.0.0.1:42051)
0811 02:41:50.609976 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:50.609984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:50.610015 (+ 31us) server_negotiation.cc:407] Connection header received
0811 02:41:50.610336 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:50.610354 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:50.610420 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:50.610681 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:50.611769 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:50.611773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:50.611774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:50.611778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:50.611824 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:50.611843 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:50.611919 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:50.612062 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 02:41:50.612135 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260811 02:41:50.612841 6700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260811 02:41:50.613014 6700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:50.613358 6699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f76c7efa3e440dcac16d4ccaf0f055e. Latest consensus state: current_term: 1 leader_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f76c7efa3e440dcac16d4ccaf0f055e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260811 02:41:50.613458 6699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:50.613875 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:50.614151 6717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:50.614691 6717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:50.616523 6717 catalog_manager.cc:1347] Generated new cluster ID: 5a53f61de93149a8a33f73a3a320bb98
I20260811 02:41:50.616580 6717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:50.618170 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:50.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:50927 (local address 127.0.0.1:42051)
0811 02:41:50.615722 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:50.615724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:50.615873 (+ 149us) server_negotiation.cc:407] Connection header received
0811 02:41:50.615976 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:50.615978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:50.615985 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:50.616042 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:50.617936 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:50.617938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:50.617939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:50.617944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:50.617962 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:50.617967 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:50.618035 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:50.618086 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 02:41:50.618102 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:41:50.623977 6717 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:50.624177 6717 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:50.628909 6717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f76c7efa3e440dcac16d4ccaf0f055e: Generated new TSK 0
I20260811 02:41:50.629127 6717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:50.639755 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6580
2026-08-11T02:41:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:50 dist-test-slave-cvwc krb5kdc[6741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:50 dist-test-slave-cvwc krb5kdc[6741](info): set up 2 sockets
Aug 11 02:41:50 dist-test-slave-cvwc krb5kdc[6741](info): commencing operation
krb5kdc: starting...
W20260811 02:41:51.856972 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:51 dist-test-slave-cvwc krb5kdc[6741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:51.988528 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:41889
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:52.077263 6765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:52.077518 6765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:52.077553 6765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:52.078770 6765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:52.078816 6765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:52.078835 6765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:52.078852 6765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:52.078868 6765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:52.080543 6765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:41889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:52.080817 6765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:52.081040 6765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:52.086344 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:52.086352 6773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:52.086452 6765 server_base.cc:1034] running on GCE node
W20260811 02:41:52.086352 6771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:52.086910 6765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:52.087199 6765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:52.088464 6765 hybrid_clock.cc:648] HybridClock initialized: now 1786416112088434 us; error 52 us; skew 500 ppm
Aug 11 02:41:52 dist-test-slave-cvwc krb5kdc[6741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416112, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:52.091440 6765 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:52.091919 6765 webserver.cc:460] Webserver started at http://127.2.251.62:42157/ using document root <none> and password file <none>
I20260811 02:41:52.092110 6765 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:52.092151 6765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:52.092247 6765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:52.093297 6765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "126b6c92db3d4912a5ff54be36eaee81"
format_stamp: "Formatted at 2026-08-11 02:41:52 on dist-test-slave-cvwc"
server_key: "58b543949da50ca3319833889234fb3b"
server_key_iv: "352bec441088fba8fd8ea6cc82bf8ebf"
server_key_version: "encryptionkey@0"
I20260811 02:41:52.093604 6765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "126b6c92db3d4912a5ff54be36eaee81"
format_stamp: "Formatted at 2026-08-11 02:41:52 on dist-test-slave-cvwc"
server_key: "58b543949da50ca3319833889234fb3b"
server_key_iv: "352bec441088fba8fd8ea6cc82bf8ebf"
server_key_version: "encryptionkey@0"
I20260811 02:41:52.094986 6765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:41:52.095762 6788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:52.095942 6765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:41:52.096033 6765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "126b6c92db3d4912a5ff54be36eaee81"
format_stamp: "Formatted at 2026-08-11 02:41:52 on dist-test-slave-cvwc"
server_key: "58b543949da50ca3319833889234fb3b"
server_key_iv: "352bec441088fba8fd8ea6cc82bf8ebf"
server_key_version: "encryptionkey@0"
I20260811 02:41:52.096097 6765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:52.118618 6765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:52.120754 6765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:52.120894 6765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:52.124873 6765 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:41889
I20260811 02:41:52.124925 6866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:41889 every 8 connection(s)
I20260811 02:41:52.125241 6765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:52.126134 6870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:52.128952 6870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81: Bootstrap starting.
I20260811 02:41:52.129527 6870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:52.129799 6870 log.cc:824] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:52.130784 6870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81: No bootstrap required, opened a new log
I20260811 02:41:52.132393 6870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126b6c92db3d4912a5ff54be36eaee81" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41889 } }
I20260811 02:41:52.132535 6870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:52.132557 6870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126b6c92db3d4912a5ff54be36eaee81, State: Initialized, Role: FOLLOWER
I20260811 02:41:52.132688 6870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126b6c92db3d4912a5ff54be36eaee81" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41889 } }
I20260811 02:41:52.132753 6870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:52.132776 6870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:52.132820 6870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:52.133670 6870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126b6c92db3d4912a5ff54be36eaee81" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41889 } }
I20260811 02:41:52.133782 6870 leader_election.cc:302] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 126b6c92db3d4912a5ff54be36eaee81; no voters:
I20260811 02:41:52.133937 6870 leader_election.cc:288] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:52.133982 6874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:52.134152 6874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [term 1 LEADER]: Becoming Leader. State: Replica: 126b6c92db3d4912a5ff54be36eaee81, State: Running, Role: LEADER
I20260811 02:41:52.134193 6870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:52.134300 6874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126b6c92db3d4912a5ff54be36eaee81" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41889 } }
I20260811 02:41:52.134688 6876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "126b6c92db3d4912a5ff54be36eaee81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126b6c92db3d4912a5ff54be36eaee81" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41889 } } }
I20260811 02:41:52.134814 6876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:52.134836 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6765
I20260811 02:41:52.134944 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:52.135116 3052 external_mini_cluster.cc:1428] Setting key 729f69beb78f26891bb219a2b81ed111
I20260811 02:41:52.135360 6885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:52.135922 6885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:52.136059 6877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 126b6c92db3d4912a5ff54be36eaee81. Latest consensus state: current_term: 1 leader_uuid: "126b6c92db3d4912a5ff54be36eaee81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126b6c92db3d4912a5ff54be36eaee81" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 41889 } } }
I20260811 02:41:52.136148 6877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:52.137615 6885 catalog_manager.cc:1347] Generated new cluster ID: b42238d8bdd24dbb81f4829dff6cd17d
I20260811 02:41:52.137666 6885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 11 02:41:52 dist-test-slave-cvwc krb5kdc[6741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:52.146184 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:52.135686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33615 (local address 127.2.251.62:41889)
0811 02:41:52.136379 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:52.136387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:52.136422 (+ 35us) server_negotiation.cc:407] Connection header received
0811 02:41:52.136594 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:52.136604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:52.136669 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:52.136802 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:52.137593 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:52.138581 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:52.139304 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:52.139719 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:52.139821 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:52.142109 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:52.142114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:52.142114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:52.142127 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:52.144151 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:52.144658 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:52.144662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:52.144663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:52.144875 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:52.145290 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:52.145292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:52.145292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:52.145551 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:52.145768 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:52.145859 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 02:41:52.145995 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":79,"threads_started":1}
I20260811 02:41:52.147145 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:52.152755 6885 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:52.153505 6885 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:52.156497 6885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 126b6c92db3d4912a5ff54be36eaee81: Generated new TSK 0
I20260811 02:41:52.156682 6885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:52.158674 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:52.152461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57119 (local address 127.2.251.62:41889)
0811 02:41:52.152500 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:52.152502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:52.152720 (+ 218us) server_negotiation.cc:407] Connection header received
0811 02:41:52.152805 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:52.152805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:52.152817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:52.152863 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:52.153443 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:52.154047 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:52.154729 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:52.154941 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:52.154996 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:52.156159 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:52.156161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:52.156161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:52.156168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:52.157558 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:52.157937 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:52.157939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:52.157939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:52.157997 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:52.158191 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:52.158192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:52.158192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:52.158299 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:52.158391 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:52.158529 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 02:41:52.158590 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 02:41:52.160853 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6765
W20260811 02:41:52.186573 6738 connection.cc:535] client connection to 127.2.251.62:41889 recv error: Network error: failed to read from TLS socket (remote: 127.2.251.62:41889): Connection reset by peer (error 104)
2026-08-11T02:41:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:52 dist-test-slave-cvwc krb5kdc[6911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:52 dist-test-slave-cvwc krb5kdc[6911](info): set up 2 sockets
Aug 11 02:41:52 dist-test-slave-cvwc krb5kdc[6911](info): commencing operation
krb5kdc: starting...
W20260811 02:41:53.387239 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:53 dist-test-slave-cvwc krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:53.516582 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39201
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:53.602764 6933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:53.603011 6933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:53.603050 6933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:53.604393 6933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:53.604449 6933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:53.604468 6933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:53.604485 6933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:53.604501 6933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:53.606225 6933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:53.606513 6933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:53.606747 6933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:53.612088 6939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:53.612112 6943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:53.612051 6945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:53.612309 6933 server_base.cc:1034] running on GCE node
I20260811 02:41:53.612581 6933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:53.612882 6933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:53.614025 6933 hybrid_clock.cc:648] HybridClock initialized: now 1786416113614006 us; error 37 us; skew 500 ppm
Aug 11 02:41:53 dist-test-slave-cvwc krb5kdc[6911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416113, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:53.619997 6933 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:53.620474 6933 webserver.cc:460] Webserver started at http://127.2.251.62:46563/ using document root <none> and password file <none>
I20260811 02:41:53.620703 6933 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:53.620754 6933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:53.620898 6933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:53.622061 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "ba21f9c085d044d6bcbb11a7d68c3be9"
format_stamp: "Formatted at 2026-08-11 02:41:53 on dist-test-slave-cvwc"
server_key: "43c670699c240244810c132ff68d7fe2"
server_key_iv: "e89c697867acb7ae2e48e00040007be6"
server_key_version: "encryptionkey@0"
I20260811 02:41:53.622444 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "ba21f9c085d044d6bcbb11a7d68c3be9"
format_stamp: "Formatted at 2026-08-11 02:41:53 on dist-test-slave-cvwc"
server_key: "43c670699c240244810c132ff68d7fe2"
server_key_iv: "e89c697867acb7ae2e48e00040007be6"
server_key_version: "encryptionkey@0"
I20260811 02:41:53.624037 6933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:53.624890 6954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:53.625078 6933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:53.625181 6933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "ba21f9c085d044d6bcbb11a7d68c3be9"
format_stamp: "Formatted at 2026-08-11 02:41:53 on dist-test-slave-cvwc"
server_key: "43c670699c240244810c132ff68d7fe2"
server_key_iv: "e89c697867acb7ae2e48e00040007be6"
server_key_version: "encryptionkey@0"
I20260811 02:41:53.625255 6933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:53.655200 6933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:53.657322 6933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:53.657451 6933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:53.661266 6933 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39201
I20260811 02:41:53.661312 7039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39201 every 8 connection(s)
I20260811 02:41:53.661664 6933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:53.662582 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:53.663223 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 6933
I20260811 02:41:53.663365 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:53.663573 3052 external_mini_cluster.cc:1428] Setting key 69ec5a43b60e286eab263905dca755c8
I20260811 02:41:53.665921 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9: Bootstrap starting.
I20260811 02:41:53.666673 7040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:53.666975 7040 log.cc:824] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:53.667692 7040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9: No bootstrap required, opened a new log
I20260811 02:41:53.669869 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:53.664154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46083 (local address 127.2.251.62:39201)
0811 02:41:53.664363 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:53.664374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:53.664412 (+ 38us) server_negotiation.cc:407] Connection header received
0811 02:41:53.664917 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:53.664926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:53.664987 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:53.665139 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:53.665411 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:53.666758 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:53.667553 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:53.668138 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:53.668267 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:53.669075 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:53.669080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:53.669080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:53.669306 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:53.669349 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:53.669364 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:53.669453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:53.669567 (+ 114us) server_negotiation.cc:299] Negotiation successful
0811 02:41:53.669598 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260811 02:41:53.670166 7040 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39201 } }
I20260811 02:41:53.670311 7040 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:53.670343 7040 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba21f9c085d044d6bcbb11a7d68c3be9, State: Initialized, Role: FOLLOWER
I20260811 02:41:53.670463 7040 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39201 } }
I20260811 02:41:53.670542 7040 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:53.670576 7040 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:53.670616 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:53.670876 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:53.671363 7040 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39201 } }
I20260811 02:41:53.671485 7040 leader_election.cc:302] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba21f9c085d044d6bcbb11a7d68c3be9; no voters:
I20260811 02:41:53.671664 7040 leader_election.cc:288] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:53.671762 7048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:53.671952 7040 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:53.671990 7048 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [term 1 LEADER]: Becoming Leader. State: Replica: ba21f9c085d044d6bcbb11a7d68c3be9, State: Running, Role: LEADER
I20260811 02:41:53.672142 7048 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39201 } }
I20260811 02:41:53.672561 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39201 } } }
I20260811 02:41:53.672580 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba21f9c085d044d6bcbb11a7d68c3be9. Latest consensus state: current_term: 1 leader_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba21f9c085d044d6bcbb11a7d68c3be9" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39201 } } }
I20260811 02:41:53.672654 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:53.672668 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:53.672881 7064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:53.673440 7064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:53.676505 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:53.672439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48703 (local address 127.2.251.62:39201)
0811 02:41:53.672476 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:53.672479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:53.673179 (+ 700us) server_negotiation.cc:407] Connection header received
0811 02:41:53.673328 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:53.673329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:53.673340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:53.673377 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:53.673598 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:53.674318 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:53.675330 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:53.675546 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:53.675610 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:53.676250 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:53.676251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:53.676251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:53.676255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:53.676269 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:53.676273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:53.676324 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:53.676421 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 02:41:53.676430 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:41:53.677204 7064 catalog_manager.cc:1347] Generated new cluster ID: 42d27ce0f72445f4b3c44716efae17e8
I20260811 02:41:53.677271 7064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:53.692778 7064 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:53.693356 7064 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:53.696341 7064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba21f9c085d044d6bcbb11a7d68c3be9: Generated new TSK 0
I20260811 02:41:53.696517 7064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:53.702880 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 6933
W20260811 02:41:53.728099 6909 connection.cc:535] client connection to 127.2.251.62:39201 recv error: Network error: failed to read from TLS socket (remote: 127.2.251.62:39201): Connection reset by peer (error 104)
2026-08-11T02:41:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:53 dist-test-slave-cvwc krb5kdc[7090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:53 dist-test-slave-cvwc krb5kdc[7090](info): set up 2 sockets
Aug 11 02:41:53 dist-test-slave-cvwc krb5kdc[7090](info): commencing operation
krb5kdc: starting...
W20260811 02:41:54.928597 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:55 dist-test-slave-cvwc krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:55.056759 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39437
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:55.142338 7115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:55.142581 7115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:55.142606 7115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:55.143775 7115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:55.143816 7115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:55.143831 7115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:55.143841 7115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:55.143852 7115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:55.145372 7115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:55.145613 7115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:55.145869 7115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:55.150837 7126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:55.150866 7115 server_base.cc:1034] running on GCE node
W20260811 02:41:55.150869 7124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:55.150837 7130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:55.151321 7115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:55.151609 7115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:55.152763 7115 hybrid_clock.cc:648] HybridClock initialized: now 1786416115152759 us; error 48 us; skew 500 ppm
Aug 11 02:41:55 dist-test-slave-cvwc krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:55.155189 7115 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:55.155606 7115 webserver.cc:460] Webserver started at http://127.2.251.62:33141/ using document root <none> and password file <none>
I20260811 02:41:55.155835 7115 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:55.155884 7115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:55.156018 7115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:55.157116 7115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "c6c679571e354f9a9e54b329a82c57c3"
format_stamp: "Formatted at 2026-08-11 02:41:55 on dist-test-slave-cvwc"
server_key: "492a67ba1418d79980632056d493abe5"
server_key_iv: "24780ded3184a391c3530984d48affe9"
server_key_version: "encryptionkey@0"
I20260811 02:41:55.157464 7115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "c6c679571e354f9a9e54b329a82c57c3"
format_stamp: "Formatted at 2026-08-11 02:41:55 on dist-test-slave-cvwc"
server_key: "492a67ba1418d79980632056d493abe5"
server_key_iv: "24780ded3184a391c3530984d48affe9"
server_key_version: "encryptionkey@0"
I20260811 02:41:55.158797 7115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260811 02:41:55.159570 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:55.159742 7115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:55.159824 7115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "c6c679571e354f9a9e54b329a82c57c3"
format_stamp: "Formatted at 2026-08-11 02:41:55 on dist-test-slave-cvwc"
server_key: "492a67ba1418d79980632056d493abe5"
server_key_iv: "24780ded3184a391c3530984d48affe9"
server_key_version: "encryptionkey@0"
I20260811 02:41:55.159894 7115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:55.163301 7115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:55.165832 7115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:55.165987 7115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:55.170011 7115 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39437
I20260811 02:41:55.170109 7231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39437 every 8 connection(s)
I20260811 02:41:55.170399 7115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:55.171411 7234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:55.172531 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7115
I20260811 02:41:55.172645 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:55.172824 3052 external_mini_cluster.cc:1428] Setting key 63004d903e32fdb3aa490a7cfeb981cf
I20260811 02:41:55.174134 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3: Bootstrap starting.
I20260811 02:41:55.174856 7234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:55.175139 7234 log.cc:824] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:55.175475 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:55.173404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49843 (local address 127.2.251.62:39437)
0811 02:41:55.173614 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:55.173624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:55.173662 (+ 38us) server_negotiation.cc:407] Connection header received
0811 02:41:55.174225 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:55.174246 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:55.174280 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:55.174431 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:55.175058 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:55.175063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:55.175063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:55.175067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:55.175108 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:55.175118 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:55.175191 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:55.175276 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 02:41:55.175319 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260811 02:41:55.175865 7234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3: No bootstrap required, opened a new log
I20260811 02:41:55.176477 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:55.177969 7234 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c679571e354f9a9e54b329a82c57c3" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39437 } }
I20260811 02:41:55.178123 7234 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:55.178148 7234 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c679571e354f9a9e54b329a82c57c3, State: Initialized, Role: FOLLOWER
I20260811 02:41:55.178270 7234 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c679571e354f9a9e54b329a82c57c3" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39437 } }
I20260811 02:41:55.178344 7234 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:55.178371 7234 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:55.178423 7234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:55.179011 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:55.177856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43177 (local address 127.2.251.62:39437)
0811 02:41:55.177890 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:55.177892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:55.178029 (+ 137us) server_negotiation.cc:407] Connection header received
0811 02:41:55.178090 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:55.178092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:55.178097 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:55.178141 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:55.178812 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:55.178813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:55.178813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:55.178818 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:55.178833 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:55.178837 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:55.178887 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:55.178935 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 02:41:55.178948 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260811 02:41:55.179123 7234 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c679571e354f9a9e54b329a82c57c3" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39437 } }
I20260811 02:41:55.179255 7234 leader_election.cc:302] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6c679571e354f9a9e54b329a82c57c3; no voters:
I20260811 02:41:55.179430 7234 leader_election.cc:288] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:55.179456 7250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:55.179653 7250 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [term 1 LEADER]: Becoming Leader. State: Replica: c6c679571e354f9a9e54b329a82c57c3, State: Running, Role: LEADER
I20260811 02:41:55.179762 7234 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:55.179781 7250 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c679571e354f9a9e54b329a82c57c3" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39437 } }
I20260811 02:41:55.180171 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6c679571e354f9a9e54b329a82c57c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c679571e354f9a9e54b329a82c57c3" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39437 } } }
I20260811 02:41:55.180210 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6c679571e354f9a9e54b329a82c57c3. Latest consensus state: current_term: 1 leader_uuid: "c6c679571e354f9a9e54b329a82c57c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6c679571e354f9a9e54b329a82c57c3" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39437 } } }
I20260811 02:41:55.180330 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:55.180425 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:55.180653 7262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:55.181290 7262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:55.183434 7262 catalog_manager.cc:1347] Generated new cluster ID: 01a2f4e596d84455877004ebcff36d89
I20260811 02:41:55.183486 7262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:55.217398 7262 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:55.217602 7262 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:55.223222 7262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6c679571e354f9a9e54b329a82c57c3: Generated new TSK 0
I20260811 02:41:55.223464 7262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:55.227370 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7115
2026-08-11T02:41:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:55 dist-test-slave-cvwc krb5kdc[7281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:55 dist-test-slave-cvwc krb5kdc[7281](info): set up 2 sockets
Aug 11 02:41:55 dist-test-slave-cvwc krb5kdc[7281](info): commencing operation
krb5kdc: starting...
W20260811 02:41:56.439670 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:56 dist-test-slave-cvwc krb5kdc[7281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:56.572820 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40087
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:34813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:56.658317 7306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:56.658578 7306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:56.658617 7306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:56.659884 7306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:56.659938 7306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:56.659956 7306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:56.659973 7306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:56.659989 7306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:56.661625 7306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:34813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:56.661926 7306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:56.662176 7306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:56.667593 7313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:56.667593 7315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:56.667735 7306 server_base.cc:1034] running on GCE node
W20260811 02:41:56.667595 7317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:56.668191 7306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:56.668547 7306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:56.669795 7306 hybrid_clock.cc:648] HybridClock initialized: now 1786416116669737 us; error 54 us; skew 500 ppm
Aug 11 02:41:56 dist-test-slave-cvwc krb5kdc[7281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:56.672745 7306 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:56.673224 7306 webserver.cc:460] Webserver started at http://127.2.251.62:40769/ using document root <none> and password file <none>
I20260811 02:41:56.673451 7306 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:56.673494 7306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:56.673594 7306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:56.674751 7306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "1f74712eb59544a7b482d317707ebcf2"
format_stamp: "Formatted at 2026-08-11 02:41:56 on dist-test-slave-cvwc"
server_key: "1dd941329c6ad88c1b33a374fde2f3af"
server_key_iv: "a454b752d4a9a8b29ab88d5e9d5fbc8b"
server_key_version: "encryptionkey@0"
I20260811 02:41:56.675081 7306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "1f74712eb59544a7b482d317707ebcf2"
format_stamp: "Formatted at 2026-08-11 02:41:56 on dist-test-slave-cvwc"
server_key: "1dd941329c6ad88c1b33a374fde2f3af"
server_key_iv: "a454b752d4a9a8b29ab88d5e9d5fbc8b"
server_key_version: "encryptionkey@0"
I20260811 02:41:56.676518 7306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:56.677364 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:56.677569 7306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:56.677664 7306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "1f74712eb59544a7b482d317707ebcf2"
format_stamp: "Formatted at 2026-08-11 02:41:56 on dist-test-slave-cvwc"
server_key: "1dd941329c6ad88c1b33a374fde2f3af"
server_key_iv: "a454b752d4a9a8b29ab88d5e9d5fbc8b"
server_key_version: "encryptionkey@0"
I20260811 02:41:56.677757 7306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:56.686372 7306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:56.688549 7306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:56.688673 7306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:56.692889 7306 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40087
I20260811 02:41:56.692925 7409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40087 every 8 connection(s)
I20260811 02:41:56.693311 7306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:56.694358 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:56.696926 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2: Bootstrap starting.
I20260811 02:41:56.697584 7411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:56.697926 7411 log.cc:824] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:56.698801 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2: No bootstrap required, opened a new log
I20260811 02:41:56.699018 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7306
I20260811 02:41:56.699155 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:56.699361 3052 external_mini_cluster.cc:1428] Setting key 37f36b18b640f2a63119895ed7c8d985
I20260811 02:41:56.701084 7411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f74712eb59544a7b482d317707ebcf2" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40087 } }
I20260811 02:41:56.701251 7411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:56.701287 7411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f74712eb59544a7b482d317707ebcf2, State: Initialized, Role: FOLLOWER
I20260811 02:41:56.701448 7411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f74712eb59544a7b482d317707ebcf2" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40087 } }
I20260811 02:41:56.701527 7411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:56.701560 7411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:56.701609 7411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:56.703378 7411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f74712eb59544a7b482d317707ebcf2" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40087 } }
I20260811 02:41:56.703513 7411 leader_election.cc:302] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f74712eb59544a7b482d317707ebcf2; no voters:
I20260811 02:41:56.703706 7411 leader_election.cc:288] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:56.703742 7416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:56.703856 7416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [term 1 LEADER]: Becoming Leader. State: Replica: 1f74712eb59544a7b482d317707ebcf2, State: Running, Role: LEADER
I20260811 02:41:56.704035 7411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:56.704003 7416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f74712eb59544a7b482d317707ebcf2" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40087 } }
I20260811 02:41:56.704754 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f74712eb59544a7b482d317707ebcf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f74712eb59544a7b482d317707ebcf2" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40087 } } }
I20260811 02:41:56.704869 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [sys.catalog]: This master's current role is: LEADER
Aug 11 02:41:56 dist-test-slave-cvwc krb5kdc[7281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:41:56.706773 7420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f74712eb59544a7b482d317707ebcf2. Latest consensus state: current_term: 1 leader_uuid: "1f74712eb59544a7b482d317707ebcf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f74712eb59544a7b482d317707ebcf2" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40087 } } }
I20260811 02:41:56.706856 7420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2 [sys.catalog]: This master's current role is: LEADER
W20260811 02:41:56.707058 7442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:41:56.707172 7442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:41:56.707286 7423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:56.707656 7423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:56.710188 7423 catalog_manager.cc:1347] Generated new cluster ID: 86b330b6d08948629db844449e5cf710
I20260811 02:41:56.710247 7423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:56.711784 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:56.699916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44559 (local address 127.2.251.62:40087)
0811 02:41:56.700090 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:56.700099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:56.700126 (+ 27us) server_negotiation.cc:407] Connection header received
0811 02:41:56.701085 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:56.701094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:56.701165 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:56.701326 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:56.702106 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:56.703098 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:56.704007 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:56.704520 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:56.704630 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:56.707422 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:56.707427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:56.707427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:56.707440 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:56.709811 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:56.710258 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:56.710262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:56.710263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:56.710389 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:56.710653 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:56.710653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:56.710654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:56.711026 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:56.711198 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:56.711402 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 02:41:56.711562 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260811 02:41:56.713078 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:56.714828 7423 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:56.715447 7423 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:56.720084 7423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f74712eb59544a7b482d317707ebcf2: Generated new TSK 0
I20260811 02:41:56.720332 7423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:56.727213 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:56.719603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41349 (local address 127.2.251.62:40087)
0811 02:41:56.719677 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:56.719679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:56.719693 (+ 14us) server_negotiation.cc:407] Connection header received
0811 02:41:56.719722 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:56.719723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:56.719748 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:56.719831 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:56.720599 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:56.721397 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:56.722248 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:56.722579 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:56.722653 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:56.724018 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:56.724020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:56.724020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:41:56.724030 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:56.725658 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:56.726053 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:56.726054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:56.726055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:56.726113 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:41:56.726379 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:41:56.726382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:41:56.726383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:41:56.726665 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:56.726862 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:56.726999 (+ 137us) server_negotiation.cc:299] Negotiation successful
0811 02:41:56.727104 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 02:41:56.729269 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7306
2026-08-11T02:41:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1502 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:56 dist-test-slave-cvwc krb5kdc[7461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:56 dist-test-slave-cvwc krb5kdc[7461](info): set up 2 sockets
Aug 11 02:41:56 dist-test-slave-cvwc krb5kdc[7461](info): commencing operation
krb5kdc: starting...
W20260811 02:41:57.953043 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:58 dist-test-slave-cvwc krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:58.086889 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:32861
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:44711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:58.172152 7487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:58.172371 7487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:58.172403 7487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:58.173518 7487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:58.173563 7487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:58.173581 7487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:58.173597 7487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:58.173614 7487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:58.175135 7487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:44711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:32861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:58.175396 7487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:58.175614 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:58.180620 7496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:58.180644 7498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:58.180630 7500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:58.180716 7487 server_base.cc:1034] running on GCE node
I20260811 02:41:58.181174 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:58.181499 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:58.182667 7487 hybrid_clock.cc:648] HybridClock initialized: now 1786416118182648 us; error 30 us; skew 500 ppm
Aug 11 02:41:58 dist-test-slave-cvwc krb5kdc[7461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416118, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:58.185137 7487 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:58.185606 7487 webserver.cc:460] Webserver started at http://127.2.251.62:34039/ using document root <none> and password file <none>
I20260811 02:41:58.185861 7487 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:58.185910 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:58.186007 7487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:58.187052 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "b2aa3e4d5b54445493712b7225b7bbbb"
format_stamp: "Formatted at 2026-08-11 02:41:58 on dist-test-slave-cvwc"
server_key: "33e49793f10860ba7893e39762bbee0e"
server_key_iv: "a47df45208b5afd3829b7f4cf281e950"
server_key_version: "encryptionkey@0"
I20260811 02:41:58.187636 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "b2aa3e4d5b54445493712b7225b7bbbb"
format_stamp: "Formatted at 2026-08-11 02:41:58 on dist-test-slave-cvwc"
server_key: "33e49793f10860ba7893e39762bbee0e"
server_key_iv: "a47df45208b5afd3829b7f4cf281e950"
server_key_version: "encryptionkey@0"
I20260811 02:41:58.189901 7487 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 02:41:58.190687 7508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:58.190862 7487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:58.190948 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "b2aa3e4d5b54445493712b7225b7bbbb"
format_stamp: "Formatted at 2026-08-11 02:41:58 on dist-test-slave-cvwc"
server_key: "33e49793f10860ba7893e39762bbee0e"
server_key_iv: "a47df45208b5afd3829b7f4cf281e950"
server_key_version: "encryptionkey@0"
I20260811 02:41:58.191009 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:58.210824 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:58.213801 7487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:58.213987 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:58.217757 7487 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:32861
I20260811 02:41:58.217803 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:32861 every 8 connection(s)
I20260811 02:41:58.218145 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:58.219297 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:58.221688 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb: Bootstrap starting.
I20260811 02:41:58.222327 7597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:58.222595 7597 log.cc:824] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:58.222800 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7487
I20260811 02:41:58.222930 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:58.223129 3052 external_mini_cluster.cc:1428] Setting key 19cebdb9db224a9052b9c9bd4891c424
I20260811 02:41:58.223417 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb: No bootstrap required, opened a new log
I20260811 02:41:58.225255 7597 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 32861 } }
I20260811 02:41:58.225404 7597 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:58.225447 7597 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2aa3e4d5b54445493712b7225b7bbbb, State: Initialized, Role: FOLLOWER
I20260811 02:41:58.225598 7597 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 32861 } }
I20260811 02:41:58.225680 7597 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:58.225735 7597 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:58.225790 7597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:58.226576 7597 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 32861 } }
I20260811 02:41:58.226678 7597 leader_election.cc:302] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2aa3e4d5b54445493712b7225b7bbbb; no voters:
I20260811 02:41:58.226866 7597 leader_election.cc:288] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:58.226904 7602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:58.227061 7602 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [term 1 LEADER]: Becoming Leader. State: Replica: b2aa3e4d5b54445493712b7225b7bbbb, State: Running, Role: LEADER
I20260811 02:41:58.227170 7597 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:58.227149 7602 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 32861 } }
I20260811 02:41:58.228500 7604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 32861 } } }
I20260811 02:41:58.228601 7604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:58.228804 7606 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2aa3e4d5b54445493712b7225b7bbbb. Latest consensus state: current_term: 1 leader_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2aa3e4d5b54445493712b7225b7bbbb" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 32861 } } }
I20260811 02:41:58.228945 7606 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:58.229099 7617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:58.229627 7617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:58.230461 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:58.223629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33799 (local address 127.2.251.62:32861)
0811 02:41:58.223886 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:58.223894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:58.223929 (+ 35us) server_negotiation.cc:407] Connection header received
0811 02:41:58.225270 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:58.225284 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:58.225342 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:58.225500 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:58.225752 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:58.227187 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:58.228055 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:58.228641 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:58.229101 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:58.229513 (+ 412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:58.229517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:58.229517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:58.229771 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:58.229812 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:58.229828 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:58.230006 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:58.230169 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 02:41:58.230235 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260811 02:41:58.231268 7617 catalog_manager.cc:1347] Generated new cluster ID: ff85bbac7797439f8e9d4473ef44c491
I20260811 02:41:58.231321 7617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:58.231428 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:58.236538 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:58.233027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40887 (local address 127.2.251.62:32861)
0811 02:41:58.233067 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:58.233070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:58.233225 (+ 155us) server_negotiation.cc:407] Connection header received
0811 02:41:58.233296 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:58.233297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:58.233309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:58.233357 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:58.233508 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:58.234252 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:58.235268 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:41:58.235546 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:41:58.235597 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:41:58.236284 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:58.236285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:58.236286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:58.236306 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:58.236323 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:58.236328 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:58.236379 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:58.236433 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 02:41:58.236458 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:41:58.237881 7617 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:58.238610 7617 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:58.245587 7617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2aa3e4d5b54445493712b7225b7bbbb: Generated new TSK 0
I20260811 02:41:58.245852 7617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:58.254289 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7487
2026-08-11T02:41:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:58 dist-test-slave-cvwc krb5kdc[7643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:58 dist-test-slave-cvwc krb5kdc[7643](info): set up 2 sockets
Aug 11 02:41:58 dist-test-slave-cvwc krb5kdc[7643](info): commencing operation
krb5kdc: starting...
W20260811 02:41:59.475764 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:41:59 dist-test-slave-cvwc krb5kdc[7643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:41:59.613643 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39439
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:39439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:41:59.701164 7666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:41:59.701411 7666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:41:59.701444 7666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:41:59.702742 7666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:41:59.702798 7666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:41:59.702817 7666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:41:59.702833 7666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:41:59.702848 7666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:41:59.704556 7666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:39439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:39439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:41:59.704850 7666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:41:59.705082 7666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:41:59.710825 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:59.710889 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:41:59.710842 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:41:59.711495 7666 server_base.cc:1034] running on GCE node
I20260811 02:41:59.711812 7666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:41:59.712121 7666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:41:59.713292 7666 hybrid_clock.cc:648] HybridClock initialized: now 1786416119713277 us; error 31 us; skew 500 ppm
Aug 11 02:41:59 dist-test-slave-cvwc krb5kdc[7643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416119, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:41:59.719470 7666 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:41:59.720017 7666 webserver.cc:460] Webserver started at http://127.2.251.62:46701/ using document root <none> and password file <none>
I20260811 02:41:59.720379 7666 fs_manager.cc:359] Metadata directory not provided
I20260811 02:41:59.720460 7666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:41:59.720587 7666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:41:59.721853 7666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "7207e4d9477141e29eebff6b2a2a86b7"
format_stamp: "Formatted at 2026-08-11 02:41:59 on dist-test-slave-cvwc"
server_key: "a0f3d73323f474662cc8cbc985daa9a2"
server_key_iv: "94becdd3593bb3901019c7791e480ad2"
server_key_version: "encryptionkey@0"
I20260811 02:41:59.722226 7666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "7207e4d9477141e29eebff6b2a2a86b7"
format_stamp: "Formatted at 2026-08-11 02:41:59 on dist-test-slave-cvwc"
server_key: "a0f3d73323f474662cc8cbc985daa9a2"
server_key_iv: "94becdd3593bb3901019c7791e480ad2"
server_key_version: "encryptionkey@0"
I20260811 02:41:59.723755 7666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 02:41:59.724848 7689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:41:59.725143 7666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:41:59.725317 7666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "7207e4d9477141e29eebff6b2a2a86b7"
format_stamp: "Formatted at 2026-08-11 02:41:59 on dist-test-slave-cvwc"
server_key: "a0f3d73323f474662cc8cbc985daa9a2"
server_key_iv: "94becdd3593bb3901019c7791e480ad2"
server_key_version: "encryptionkey@0"
I20260811 02:41:59.725430 7666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:41:59.729768 7666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:41:59.732918 7666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:41:59.733111 7666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:41:59.737973 7666 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:39439
I20260811 02:41:59.738009 7778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:39439 every 8 connection(s)
I20260811 02:41:59.738391 7666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:41:59.739509 7780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:41:59.740758 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7666
I20260811 02:41:59.740873 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:41:59.741040 3052 external_mini_cluster.cc:1428] Setting key 8ad9fd1909de5e4c06e2e1e3aff08388
I20260811 02:41:59.743678 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7: Bootstrap starting.
I20260811 02:41:59.744051 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:59.741746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47985 (local address 127.2.251.62:39439)
0811 02:41:59.742058 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:59.742067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:59.742100 (+ 33us) server_negotiation.cc:407] Connection header received
0811 02:41:59.742657 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:59.742665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:59.742694 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:59.742843 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:59.743590 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:59.743595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:59.743595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:59.743600 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:59.743639 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:59.743649 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:59.743710 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:59.743815 (+ 105us) server_negotiation.cc:299] Negotiation successful
0811 02:41:59.743860 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
I20260811 02:41:59.744488 7780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7: Neither blocks nor log segments found. Creating new log.
I20260811 02:41:59.744771 7780 log.cc:824] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7: Log is configured to *not* fsync() on all Append() calls
I20260811 02:41:59.745604 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:41:59.745678 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7: No bootstrap required, opened a new log
I20260811 02:41:59.748171 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:41:59.747178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58587 (local address 127.2.251.62:39439)
0811 02:41:59.747307 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 02:41:59.747309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:41:59.747316 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:41:59.747338 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:41:59.747339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:41:59.747353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:41:59.747398 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:41:59.747970 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:41:59.747971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:41:59.747971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 02:41:59.747975 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:41:59.747991 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 02:41:59.747995 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:41:59.748042 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 02:41:59.748088 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 02:41:59.748103 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260811 02:41:59.748703 7780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7207e4d9477141e29eebff6b2a2a86b7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39439 } }
I20260811 02:41:59.749038 7780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 02:41:59.749099 7780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7207e4d9477141e29eebff6b2a2a86b7, State: Initialized, Role: FOLLOWER
I20260811 02:41:59.749226 7780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7207e4d9477141e29eebff6b2a2a86b7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39439 } }
I20260811 02:41:59.749310 7780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 02:41:59.749336 7780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 02:41:59.749369 7780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:41:59.750259 7780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7207e4d9477141e29eebff6b2a2a86b7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39439 } }
I20260811 02:41:59.750401 7780 leader_election.cc:302] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7207e4d9477141e29eebff6b2a2a86b7; no voters:
I20260811 02:41:59.750739 7780 leader_election.cc:288] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 02:41:59.750795 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:41:59.750991 7794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7207e4d9477141e29eebff6b2a2a86b7, State: Running, Role: LEADER
I20260811 02:41:59.751122 7780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:41:59.751139 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7207e4d9477141e29eebff6b2a2a86b7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39439 } }
I20260811 02:41:59.751919 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7207e4d9477141e29eebff6b2a2a86b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7207e4d9477141e29eebff6b2a2a86b7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39439 } } }
I20260811 02:41:59.752028 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:59.752262 7812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:41:59.752399 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7207e4d9477141e29eebff6b2a2a86b7. Latest consensus state: current_term: 1 leader_uuid: "7207e4d9477141e29eebff6b2a2a86b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7207e4d9477141e29eebff6b2a2a86b7" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 39439 } } }
I20260811 02:41:59.752454 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7 [sys.catalog]: This master's current role is: LEADER
I20260811 02:41:59.752786 7812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:41:59.754632 7812 catalog_manager.cc:1347] Generated new cluster ID: 90e13aeb672e4c97a3dacf7ed0cb963b
I20260811 02:41:59.754684 7812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:41:59.771399 7812 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:41:59.771595 7812 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:41:59.777676 7812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7207e4d9477141e29eebff6b2a2a86b7: Generated new TSK 0
I20260811 02:41:59.777940 7812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:41:59.788988 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7666
2026-08-11T02:41:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1527 ms)
[----------] 18 tests from AuthTokenIssuingTest (27604 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:41:59 dist-test-slave-cvwc krb5kdc[7831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:41:59 dist-test-slave-cvwc krb5kdc[7831](info): set up 2 sockets
Aug 11 02:41:59 dist-test-slave-cvwc krb5kdc[7831](info): commencing operation
krb5kdc: starting...
W20260811 02:42:00.991825 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:42:01.135134 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40815
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40815,127.2.251.61:41565,127.2.251.60:37445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:42:01.223124 7849 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 02:42:01.223369 7849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:42:01.223389 7849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:42:01.223412 7849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:42:01.224628 7849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:42:01.224681 7849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:42:01.224695 7849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:42:01.224706 7849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:42:01.224717 7849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:42:01.227017 7849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40815,127.2.251.61:41565,127.2.251.60:37445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:40815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:42:01.227300 7849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:42:01.227636 7849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:42:01.233623 7854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:01.233649 7855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:01.233834 7860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:01.233932 7849 server_base.cc:1034] running on GCE node
I20260811 02:42:01.234192 7849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:42:01.234510 7849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:42:01.235689 7849 hybrid_clock.cc:648] HybridClock initialized: now 1786416121235666 us; error 34 us; skew 500 ppm
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:42:01.238411 7849 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:42:01.238960 7849 webserver.cc:460] Webserver started at http://127.2.251.62:43285/ using document root <none> and password file <none>
I20260811 02:42:01.239181 7849 fs_manager.cc:359] Metadata directory not provided
I20260811 02:42:01.239231 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:42:01.239346 7849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:42:01.240677 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "79319316a598423887b95a8be62c94b4"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "4b2d468a76d02924f96682ddaf4e77fe"
server_key_iv: "6c11cb7099fe2ed28cee25ec89f4f82f"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.241088 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "79319316a598423887b95a8be62c94b4"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "4b2d468a76d02924f96682ddaf4e77fe"
server_key_iv: "6c11cb7099fe2ed28cee25ec89f4f82f"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.242547 7849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:42:01.243362 7871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:42:01.243669 7849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:42:01.243779 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "79319316a598423887b95a8be62c94b4"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "4b2d468a76d02924f96682ddaf4e77fe"
server_key_iv: "6c11cb7099fe2ed28cee25ec89f4f82f"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.243863 7849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:42:01.261703 7849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:42:01.263939 7849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:42:01.264067 7849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:42:01.268291 7849 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:40815
I20260811 02:42:01.268355 7953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:40815 every 8 connection(s)
I20260811 02:42:01.268713 7849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:42:01.269850 7958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:42:01.271469 7958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.271814 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7849
I20260811 02:42:01.271958 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:42:01.272310 3052 external_mini_cluster.cc:1428] Setting key 61076ca05cfa030ed34ca8f785645dd4
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.251.61@KRBTEST.COM; defaulting to no policy
I20260811 02:42:01.283224 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.272297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40815 (local address 127.0.0.1:51738)
0811 02:42:01.272499 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.272550 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.273057 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.273471 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.273477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.274057 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.274239 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.274248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.275697 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.275698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.276487 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.276490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.277028 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.277029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.277166 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.277691 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.277700 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.279428 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.281653 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.281662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.281665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.281928 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.282173 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.282174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.282174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.282279 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.282670 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.282680 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.282757 (+ 77us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.282855 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.283009 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260811 02:42:01.283231 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.272500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.2.251.62:40815)
0811 02:42:01.272645 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.272650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.272695 (+ 45us) server_negotiation.cc:407] Connection header received
0811 02:42:01.273215 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.273217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.273241 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.273354 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.274375 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.275589 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.276580 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.276928 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.277034 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.279531 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.279541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.279542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.279552 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.281564 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.282019 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.282022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.282022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.282110 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.282336 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.282336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.282336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.282580 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.282646 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.282916 (+ 270us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.283080 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260811 02:42:01.284061 7958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.284498 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.284320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46020)
0811 02:42:01.284414 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.284469 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260811 02:42:01.284680 7873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.61@KRBTEST.COM" created.
I20260811 02:42:01.285163 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.285066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46032)
0811 02:42:01.285108 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.285139 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:42:01.285267 7958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:41565: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab.
Entry for principal kudu/127.2.251.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab.
I20260811 02:42:01.300916 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:41565
--webserver_interface=127.2.251.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab
--principal=kudu/127.2.251.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40815,127.2.251.61:41565,127.2.251.60:37445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:42:01.334581 7958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } attempt: 1
I20260811 02:42:01.335389 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.335140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46034)
0811 02:42:01.335258 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.335324 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 02:42:01.335850 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.335755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46044)
0811 02:42:01.335787 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.335816 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260811 02:42:01.335958 7958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:41565: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
W20260811 02:42:01.390885 7970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 02:42:01.391146 7970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:42:01.391173 7970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:42:01.391201 7970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:42:01.392452 7970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:42:01.392501 7970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:42:01.392519 7970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:42:01.392536 7970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:42:01.392551 7970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:42:01.394315 7970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40815,127.2.251.61:41565,127.2.251.60:37445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab
--principal=kudu/127.2.251.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.251.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:42:01.394601 7970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:42:01.394824 7970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:42:01.399698 7981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:01.399693 7975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:01.399689 7976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:01.399969 7970 server_base.cc:1034] running on GCE node
I20260811 02:42:01.400226 7970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:42:01.400523 7970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:42:01.401690 7970 hybrid_clock.cc:648] HybridClock initialized: now 1786416121401674 us; error 32 us; skew 500 ppm
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:42:01.404124 7958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } attempt: 2
I20260811 02:42:01.404652 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.404493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46058)
0811 02:42:01.404573 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.404616 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 02:42:01.404691 7970 init.cc:377] Logged in from keytab as kudu/127.2.251.61@KRBTEST.COM (short username kudu)
I20260811 02:42:01.406486 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.406334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46070)
0811 02:42:01.406414 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.406456 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 02:42:01.406638 7958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:41565: Network error: Client connection negotiation failed: client connection to 127.2.251.61:41565: connect: Connection refused (error 111)
I20260811 02:42:01.406945 7970 webserver.cc:460] Webserver started at http://127.2.251.61:38235/ using document root <none> and password file <none>
I20260811 02:42:01.407398 7970 fs_manager.cc:359] Metadata directory not provided
I20260811 02:42:01.407471 7970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:42:01.407649 7970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:42:01.409013 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data/instance:
uuid: "4ce15c84e41c4c8991cbe0978520a1ea"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "7ee6621c858d76fb3514153dcb67a632"
server_key_iv: "24de948c248bee778df8caf75a925245"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.409386 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal/instance:
uuid: "4ce15c84e41c4c8991cbe0978520a1ea"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "7ee6621c858d76fb3514153dcb67a632"
server_key_iv: "24de948c248bee778df8caf75a925245"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.411001 7970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260811 02:42:01.412165 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:42:01.412407 7970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:42:01.412514 7970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal
uuid: "4ce15c84e41c4c8991cbe0978520a1ea"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "7ee6621c858d76fb3514153dcb67a632"
server_key_iv: "24de948c248bee778df8caf75a925245"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.412593 7970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:42:01.422645 7970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:42:01.424688 7970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:42:01.424832 7970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:42:01.430625 7970 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.61:41565
I20260811 02:42:01.430671 8077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.61:41565 every 8 connection(s)
I20260811 02:42:01.431093 7970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
I20260811 02:42:01.432137 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:42:01.433806 8079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.437574 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 7970
I20260811 02:42:01.437685 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-1/wal/instance
I20260811 02:42:01.437935 3052 external_mini_cluster.cc:1428] Setting key 54cc4836afa75cd11f3e3f17e14d8c18
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:42:01.445240 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.434609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.2.251.62:40815)
0811 02:42:01.434677 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.434680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.435076 (+ 396us) server_negotiation.cc:407] Connection header received
0811 02:42:01.435765 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.435767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.435781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.435861 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.436681 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.437203 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.438755 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.438985 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.439051 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.441955 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.441957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.441958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.441967 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.443648 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.444145 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.444147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.444147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.444218 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.444464 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.444464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.444465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.444615 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.444689 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.445026 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.445125 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 02:42:01.445344 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.434559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40815 (local address 127.0.0.1:51740)
0811 02:42:01.434832 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.434896 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.435627 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.436023 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.436028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.436312 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.436552 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.436558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.437350 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.437352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.438635 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.438639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.439099 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.439100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.439246 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.439873 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.439883 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.441830 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.443756 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.443764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.443766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.444068 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.444298 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.444299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.444299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.444408 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.444725 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.444732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.444886 (+ 154us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.444993 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.445139 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260811 02:42:01.446053 8079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.251.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.60@KRBTEST.COM" created.
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:42:01.453694 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.446308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46084)
0811 02:42:01.446355 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.446366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.446409 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.446638 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.446639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.446796 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.446848 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.446850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.447717 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.447718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.448330 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.448332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.448771 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.448772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.448834 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.449220 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.449225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.450768 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.452542 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.452544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.452544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.452790 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.452992 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.452993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.452994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.453063 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.453317 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.453318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.453372 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.453453 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.453564 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 02:42:01.454039 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.446380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.2.251.61:41565)
0811 02:42:01.446525 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.446528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.446543 (+ 15us) server_negotiation.cc:407] Connection header received
0811 02:42:01.446558 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.446559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.446569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.446624 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.446915 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.447643 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.448403 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.448701 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.448773 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.450853 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.450863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.450864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.450872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.452461 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.452876 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.452879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.452879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.452932 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.453119 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.453120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.453120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.453258 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.453315 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.453516 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.453632 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260811 02:42:01.454627 8079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.455049 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.454916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:44964)
0811 02:42:01.454967 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.455016 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 02:42:01.455199 8010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
I20260811 02:42:01.455642 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.455561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:44976)
0811 02:42:01.455602 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.455627 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:42:01.455749 8079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:37445: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab.
Entry for principal kudu/127.2.251.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab.
I20260811 02:42:01.468360 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:37445
--webserver_interface=127.2.251.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab
--principal=kudu/127.2.251.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:40815,127.2.251.61:41565,127.2.251.60:37445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:42:01.497816 7958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } attempt: 3
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:42:01.505014 8079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } attempt: 1
I20260811 02:42:01.505842 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.505698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:44984)
0811 02:42:01.505767 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.505802 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 02:42:01.506086 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.498285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.2.251.61:41565)
0811 02:42:01.498359 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.498361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.498371 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:01.498401 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.498402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.498412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.498471 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.498807 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.499309 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.500219 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.500461 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.500526 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.503000 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.503002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.503002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.503009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.504876 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.505277 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.505278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.505279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.505334 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.505517 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.505518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.505519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.505643 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.505698 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.505904 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.506003 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 02:42:01.506157 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.506079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:44996)
0811 02:42:01.506113 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.506135 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260811 02:42:01.506232 8079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:37445: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
I20260811 02:42:01.506394 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.498199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46092)
0811 02:42:01.498262 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.498274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.498318 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.498490 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.498491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.498659 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.498730 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.498732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.499387 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.499389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.500110 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.500114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.500529 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.500530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.500606 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.501229 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.501234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.502883 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.504982 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.504983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.504984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.505174 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.505397 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.505398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.505398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.505459 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.505701 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.505702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.505783 (+ 81us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.505855 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.506068 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260811 02:42:01.506886 7958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.507197 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.507115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45010)
0811 02:42:01.507150 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.507178 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260811 02:42:01.507498 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.507416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45022)
0811 02:42:01.507454 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.507477 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 02:42:01.507582 7958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:37445: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
I20260811 02:42:01.538748 7958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } attempt: 1
I20260811 02:42:01.539263 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.539139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45028)
0811 02:42:01.539199 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.539232 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 02:42:01.539588 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.539506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45042)
0811 02:42:01.539547 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.539570 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260811 02:42:01.539667 7958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:37445: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
W20260811 02:42:01.560715 8093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 02:42:01.560979 8093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:42:01.561007 8093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:42:01.561035 8093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:42:01.562350 8093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:42:01.562417 8093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:42:01.562436 8093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:42:01.562453 8093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:42:01.562469 8093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:42:01.564388 8093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:40815,127.2.251.61:41565,127.2.251.60:37445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab
--principal=kudu/127.2.251.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:37445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.251.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:42:01.564716 8093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:42:01.564978 8093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:42:01.570226 8093 server_base.cc:1034] running on GCE node
W20260811 02:42:01.570214 8103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:01.570199 8105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:01.570205 8102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:01.570719 8093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:42:01.571022 8093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:42:01.572196 8093 hybrid_clock.cc:648] HybridClock initialized: now 1786416121572171 us; error 38 us; skew 500 ppm
I20260811 02:42:01.574368 8079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } attempt: 2
I20260811 02:42:01.574939 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.574777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45058)
0811 02:42:01.574855 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.574899 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:42:01.575300 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.575199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45070)
0811 02:42:01.575235 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.575272 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260811 02:42:01.575387 8079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:37445: Network error: Client connection negotiation failed: client connection to 127.2.251.60:37445: connect: Connection refused (error 111)
I20260811 02:42:01.575580 8093 init.cc:377] Logged in from keytab as kudu/127.2.251.60@KRBTEST.COM (short username kudu)
I20260811 02:42:01.576115 8093 webserver.cc:460] Webserver started at http://127.2.251.60:44409/ using document root <none> and password file <none>
I20260811 02:42:01.576383 8093 fs_manager.cc:359] Metadata directory not provided
I20260811 02:42:01.576438 8093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:42:01.576587 8093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:42:01.577785 8093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data/instance:
uuid: "f2278344e3294fdbbdad3260c9f1f9cb"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "ca1440d3835e7ce93ac867ea51f81d3f"
server_key_iv: "48b28cc861234b0e2e2b481d828cbe52"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.578162 8093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal/instance:
uuid: "f2278344e3294fdbbdad3260c9f1f9cb"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "ca1440d3835e7ce93ac867ea51f81d3f"
server_key_iv: "48b28cc861234b0e2e2b481d828cbe52"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.579612 8093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260811 02:42:01.580477 8115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:42:01.580693 8093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:42:01.580792 8093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal
uuid: "f2278344e3294fdbbdad3260c9f1f9cb"
format_stamp: "Formatted at 2026-08-11 02:42:01 on dist-test-slave-cvwc"
server_key: "ca1440d3835e7ce93ac867ea51f81d3f"
server_key_iv: "48b28cc861234b0e2e2b481d828cbe52"
server_key_version: "encryptionkey@0"
I20260811 02:42:01.580868 8093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:42:01.590654 8093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:42:01.593526 8093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:42:01.593924 8093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:42:01.599017 8093 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.60:37445
I20260811 02:42:01.599107 8202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.60:37445 every 8 connection(s)
I20260811 02:42:01.599515 8093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
I20260811 02:42:01.600649 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:42:01.602428 8203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.605686 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8093
I20260811 02:42:01.605875 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786415831315317-3052-0/minicluster-data/master-2/wal/instance
I20260811 02:42:01.606092 3052 external_mini_cluster.cc:1428] Setting key e03e6af9a97456c310e24dc07bd23715
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:42:01.614817 7958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } attempt: 2
I20260811 02:42:01.616508 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.603358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.2.251.62:40815)
0811 02:42:01.603461 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.603465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.603798 (+ 333us) server_negotiation.cc:407] Connection header received
0811 02:42:01.605424 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.605425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.605444 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.605513 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.606927 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.607467 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.609071 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.609279 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.609370 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.612707 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.612710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.612710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.612721 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.614400 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.614916 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.614918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.614918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.614990 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.615488 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.615490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.615490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.615657 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.615694 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.616333 (+ 639us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.616428 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 02:42:01.617121 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.603235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:40815 (local address 127.0.0.1:51744)
0811 02:42:01.603617 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.603671 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.605212 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.605658 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.605664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.606412 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.606762 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.606775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.607578 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.607580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.608935 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.608939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.609407 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.609408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.609553 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.610181 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.610194 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.612524 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.614522 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.614530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.614534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.614818 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.615248 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.615249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.615250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.615396 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.615733 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.615739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.615882 (+ 143us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.616294 (+ 412us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.616579 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260811 02:42:01.617401 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.606647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.2.251.60:37445)
0811 02:42:01.606963 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.606969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.607029 (+ 60us) server_negotiation.cc:407] Connection header received
0811 02:42:01.607117 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.607121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.607150 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.607220 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.607905 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.608733 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.609499 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.609860 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.609918 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.613203 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.613221 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.613223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.613236 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.615282 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.615770 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.615774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.615774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.615846 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.616050 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.616051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.616051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.616210 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.616373 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.617100 (+ 727us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.617207 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
I20260811 02:42:01.617806 8203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:42:01.624857 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.615134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45094)
0811 02:42:01.615402 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.615415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.615458 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.615628 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.615629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.615779 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.615837 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.615840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.617484 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.617485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.618218 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.618222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.618708 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.618709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.618765 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.619285 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.619290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.620811 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.623631 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.623633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.623633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.623906 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.624129 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.624131 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.624132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.624199 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.624489 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.624490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.624545 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.624620 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.624747 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260811 02:42:01.625279 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.615223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.2.251.60:37445)
0811 02:42:01.615428 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.615430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.615483 (+ 53us) server_negotiation.cc:407] Connection header received
0811 02:42:01.615505 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.615506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.615518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.615591 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.616189 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.617382 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.618321 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.618655 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.618714 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.621277 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.621279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.621279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.621285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.623527 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.623987 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.623988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.623989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.624057 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.624271 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.624272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.624272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.624421 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.624487 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.624924 (+ 437us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.625110 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":111,"threads_started":1}
I20260811 02:42:01.626392 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.618182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.2.251.61:41565)
0811 02:42:01.618259 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.618261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.618271 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:01.618381 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.618382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.618394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.618444 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.618891 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.619927 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.620633 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.620854 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.620915 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.623285 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.623287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.623287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.623297 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.625066 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.625509 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.625510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.625511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.625573 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.625816 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.625817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.625818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.625962 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.626036 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.626229 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.626310 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 02:42:01.626448 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.618483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.2.251.62:40815)
0811 02:42:01.618519 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.618520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.618527 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:42:01.618592 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.618592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.618602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.618644 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.619143 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.619880 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.620896 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.621149 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.621218 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.623722 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.623724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.623725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.623732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.625268 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.625660 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.625661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.625661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.625737 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.625943 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.625944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.625945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.626072 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.626218 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.626278 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.626364 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:42:01.626668 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.618103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:41565 (local address 127.0.0.1:46096)
0811 02:42:01.618231 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.618242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.618281 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.618511 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.618512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.618685 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.618761 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.618763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.620013 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.620014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.620528 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.620531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.620971 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.620972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.621051 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.621551 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.621555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.623175 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.625147 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.625148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.625149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.625420 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.625640 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.625641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.625642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.625705 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.626044 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.626046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.626096 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.626476 (+ 380us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.626581 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260811 02:42:01.627381 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4: Bootstrap starting.
I20260811 02:42:01.628042 7958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4: Neither blocks nor log segments found. Creating new log.
I20260811 02:42:01.628304 7958 log.cc:824] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4: Log is configured to *not* fsync() on all Append() calls
I20260811 02:42:01.628858 8203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:01.629129 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4: No bootstrap required, opened a new log
I20260811 02:42:01.631259 7958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.631433 7958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79319316a598423887b95a8be62c94b4, State: Initialized, Role: FOLLOWER
I20260811 02:42:01.631522 7958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.631843 8213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.632009 8213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.631916 7958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: configured and running, proceeding with master startup.
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
W20260811 02:42:01.634706 8227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:42:01.634764 8227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:42:01.636898 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.629138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45110)
0811 02:42:01.629173 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.629184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.629222 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.629520 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.629521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.629666 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.629699 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.629701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.630470 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.630471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.630985 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.630987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.631340 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.631341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.631388 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.631870 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.631875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.633416 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.635363 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.635364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.635364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.635578 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.635803 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.635804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.635805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.635858 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.636533 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.636534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.636580 (+ 46us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.636651 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.636785 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260811 02:42:01.637183 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.629235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.2.251.60:37445)
0811 02:42:01.629433 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.629434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.629440 (+ 6us) server_negotiation.cc:407] Connection header received
0811 02:42:01.629454 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.629455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.629462 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.629502 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.629792 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.630371 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.631064 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.631274 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.631339 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.633491 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.633493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.633493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.633501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.635261 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.635659 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.635660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.635661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.635735 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.635922 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.635922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.635923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.636420 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.636505 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.637028 (+ 523us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.637104 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260811 02:42:01.638691 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.629139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.2.251.61:41565)
0811 02:42:01.629181 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.629183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.629393 (+ 210us) server_negotiation.cc:407] Connection header received
0811 02:42:01.629437 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.629437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.629448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.629492 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.630369 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.631035 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.632252 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.632462 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.632527 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.635711 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.635713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.635713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.635723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.637418 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.637878 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.637880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.637880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.637947 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.638144 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.638144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.638145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.638277 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.638426 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.638516 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.638599 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:42:01.635697 7849 master_runner.cc:186] Error getting master registration for 127.2.251.61:41565: OK, instance_id { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" instance_seqno: 1786416121428409 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 02:42:01.639989 7849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:42:01.640336 8203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: Bootstrap starting.
I20260811 02:42:01.640849 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:42:01.641129 8203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: Neither blocks nor log segments found. Creating new log.
I20260811 02:42:01.641399 8203 log.cc:824] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: Log is configured to *not* fsync() on all Append() calls
I20260811 02:42:01.641948 7912 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "79319316a598423887b95a8be62c94b4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51752
I20260811 02:42:01.642118 7912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 02:42:01.642177 7912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:42:01.642246 8203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: No bootstrap required, opened a new log
I20260811 02:42:01.643139 7912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.643437 7912 leader_election.cc:288] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [CANDIDATE]: Term 1 election: Requested vote from peers 4ce15c84e41c4c8991cbe0978520a1ea (127.2.251.61:41565), f2278344e3294fdbbdad3260c9f1f9cb (127.2.251.60:37445)
I20260811 02:42:01.644011 8203 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.644172 8203 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2278344e3294fdbbdad3260c9f1f9cb, State: Initialized, Role: FOLLOWER
I20260811 02:42:01.644259 8203 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.644769 8203 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:42:01.644804 8159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79319316a598423887b95a8be62c94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2278344e3294fdbbdad3260c9f1f9cb"
I20260811 02:42:01.644909 8159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:42:01.645921 8159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79319316a598423887b95a8be62c94b4 in term 1.
I20260811 02:42:01.646214 7878 leader_election.cc:302] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79319316a598423887b95a8be62c94b4, f2278344e3294fdbbdad3260c9f1f9cb; no voters:
I20260811 02:42:01.646469 8213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:42:01.646670 8213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [term 1 LEADER]: Becoming Leader. State: Replica: 79319316a598423887b95a8be62c94b4, State: Running, Role: LEADER
I20260811 02:42:01.646788 8213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
W20260811 02:42:01.647977 8256 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:42:01.648036 8256 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:42:01.649509 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.649611 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.649389 8093 master_runner.cc:186] Error getting master registration for 127.2.251.61:41565: OK, instance_id { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" instance_seqno: 1786416121428409 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 02:42:01.651862 8093 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:42:01.652653 8259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79319316a598423887b95a8be62c94b4. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.653167 8259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: This master's current role is: LEADER
I20260811 02:42:01.653903 8267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:42:01.654189 8267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:42:01.656122 8159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [term 1 FOLLOWER]: Refusing update from remote peer 79319316a598423887b95a8be62c94b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:42:01.656486 8213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:42:01.658128 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79319316a598423887b95a8be62c94b4. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.658252 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.658672 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.644998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.2.251.60:37445)
0811 02:42:01.645043 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.645046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.645199 (+ 153us) server_negotiation.cc:407] Connection header received
0811 02:42:01.645304 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.645305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.645318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.645350 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.646670 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.647387 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.648473 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.648773 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.648826 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.655306 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.655309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.655309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.655334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.657153 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.657756 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.657758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.657759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.657839 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.658041 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.658041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.658042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.658179 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.658262 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.658486 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.658577 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:42:01.660581 8259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.660697 8259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: This master's current role is: LEADER
I20260811 02:42:01.661113 8235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.661204 8235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.661574 8269 mvcc.cc:205] Tried to move back new op lower bound from 7317160434300301312 to 7317160434266353664. Current Snapshot: MvccSnapshot[applied={T|T < 7317160434300301312}]
I20260811 02:42:01.662446 8270 mvcc.cc:205] Tried to move back new op lower bound from 7317160434300301312 to 7317160434266353664. Current Snapshot: MvccSnapshot[applied={T|T < 7317160434300301312}]
I20260811 02:42:01.663480 8267 catalog_manager.cc:1347] Generated new cluster ID: 443802e4f426498d818956ee1621e110
I20260811 02:42:01.663533 8267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:42:01.666503 8079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } attempt: 3
I20260811 02:42:01.666824 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.646489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.2.251.62:40815)
0811 02:42:01.646543 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.646546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.655552 (+ 9006us) server_negotiation.cc:407] Connection header received
0811 02:42:01.655705 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.655706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.655718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.655748 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.657241 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.657984 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.659962 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.660263 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.660333 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.663395 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.663397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.663397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.663406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.664915 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.665617 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.665631 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.665631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.665700 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.666054 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.666055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.666056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.666202 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.666274 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.666629 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.666727 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 11 02:42:01 dist-test-slave-cvwc krb5kdc[7831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416121, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:42:01.674233 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.666774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:37445 (local address 127.0.0.1:45128)
0811 02:42:01.666861 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:01.666872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:01.666903 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:01.667190 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:01.667191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:01.667361 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:01.667406 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.667407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.668270 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.668271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.669035 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:01.669037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.669518 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.669519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:01.669571 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.670050 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:01.670055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:01.671503 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:01.673200 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.673201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.673202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.673420 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.673588 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:01.673589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:01.673589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:01.673635 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:01.673916 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:01.673916 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:01.673963 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 02:42:01.674021 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 02:42:01.674128 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 02:42:01.674600 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.666989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.2.251.60:37445)
0811 02:42:01.667069 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.667070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.667077 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:42:01.667099 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.667100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.667109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.667149 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.667470 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.668187 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.669117 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.669412 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.669465 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.671607 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.671609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.671609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.671618 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.673105 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.673485 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.673487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.673487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.673535 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.673683 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.673684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.673684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.673829 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.673888 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.674435 (+ 547us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.674513 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 02:42:01.676131 8079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: Bootstrap starting.
I20260811 02:42:01.676877 8079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: Neither blocks nor log segments found. Creating new log.
I20260811 02:42:01.677172 8079 log.cc:824] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: Log is configured to *not* fsync() on all Append() calls
I20260811 02:42:01.681191 8267 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:42:01.681486 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.649370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.2.251.61:41565)
0811 02:42:01.649419 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.649421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.657668 (+ 8247us) server_negotiation.cc:407] Connection header received
0811 02:42:01.657731 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.657733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.657748 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.657809 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.659568 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.660227 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.662685 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.662989 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.663059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.678429 (+ 15370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:01.678432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:01.678432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:01.678442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:01.680080 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.680556 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.680558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.680558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.680617 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:01.680827 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:01.680828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:01.680828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:01.680965 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:01.681056 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.681293 (+ 237us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.681386 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:42:01.682798 8267 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:42:01.690655 8079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: No bootstrap required, opened a new log
I20260811 02:42:01.692888 8079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.693073 8079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce15c84e41c4c8991cbe0978520a1ea, State: Initialized, Role: FOLLOWER
I20260811 02:42:01.693157 8079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } }
I20260811 02:42:01.693576 8079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:42:01.695052 8038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:42:01.696120 8038 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [term 1 FOLLOWER]: Refusing update from remote peer 79319316a598423887b95a8be62c94b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:42:01.696475 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79319316a598423887b95a8be62c94b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ce15c84e41c4c8991cbe0978520a1ea"
I20260811 02:42:01.697361 8042 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79319316a598423887b95a8be62c94b4 in term 1.
I20260811 02:42:01.698797 8213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:42:01.700505 8267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4: Generated new TSK 0
I20260811 02:42:01.700735 8267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 02:42:01.701251 8275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.701537 8275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.701645 8275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79319316a598423887b95a8be62c94b4. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.701797 8275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.701872 8275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:01.701964 8275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:01.703998 8299 mvcc.cc:205] Tried to move back new op lower bound from 7317160434466480128 to 7317160434266353664. Current Snapshot: MvccSnapshot[applied={T|T < 7317160434300301312}]
I20260811 02:42:01.706519 8297 catalog_manager.cc:1259] Loaded cluster ID: 443802e4f426498d818956ee1621e110
I20260811 02:42:01.706630 8297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: loading cluster ID for follower catalog manager: success
I20260811 02:42:01.707886 8297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: acquiring CA information for follower catalog manager: success
I20260811 02:42:01.709345 8297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ce15c84e41c4c8991cbe0978520a1ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 02:42:01.728399 8089 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.726784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.2.251.61:41565)
0811 02:42:01.726829 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.726830 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.727605 (+ 775us) server_negotiation.cc:407] Connection header received
0811 02:42:01.727716 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.727718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.727730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.727772 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.728314 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46134: BlockingRecv error: recv got EOF from 127.0.0.1:46134 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:01.729312 7962 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.726517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.2.251.62:40815)
0811 02:42:01.726574 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.726576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.728715 (+ 2139us) server_negotiation.cc:407] Connection header received
0811 02:42:01.728847 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.728849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.728860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.728892 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.729244 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51766: BlockingRecv error: recv got EOF from 127.0.0.1:51766 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260811 02:42:01.730134 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.727037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.2.251.60:37445)
0811 02:42:01.727079 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.727080 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.729567 (+ 2487us) server_negotiation.cc:407] Connection header received
0811 02:42:01.729664 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.729665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.729675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.729704 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.730072 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45138: BlockingRecv error: recv got EOF from 127.0.0.1:45138 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:42:01.735913 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.732063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.2.251.61:41565)
0811 02:42:01.732102 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.732104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.732357 (+ 253us) server_negotiation.cc:407] Connection header received
0811 02:42:01.732524 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.732525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.732579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.732631 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.732902 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.733688 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.734938 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.735218 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.735259 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.735485 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.735628 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.735681 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.735808 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.735832 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260811 02:42:01.742727 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.738779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.2.251.60:37445)
0811 02:42:01.738852 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.738854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.742004 (+ 3150us) server_negotiation.cc:407] Connection header received
0811 02:42:01.742138 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.742140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.742163 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.742195 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.742654 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45152: BlockingRecv error: recv got EOF from 127.0.0.1:45152 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 02:42:01.749060 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.745388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.2.251.61:41565)
0811 02:42:01.745434 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.745437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.745766 (+ 329us) server_negotiation.cc:407] Connection header received
0811 02:42:01.745930 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.745931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.745944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.745974 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.746256 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.747050 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.748230 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.748525 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.748566 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.748784 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.748822 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.748848 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.748944 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.748970 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260811 02:42:01.752774 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.751922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.2.251.60:37445)
0811 02:42:01.751966 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.751968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.752184 (+ 216us) server_negotiation.cc:407] Connection header received
0811 02:42:01.752279 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.752280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.752292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.752320 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.752703 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45154: BlockingRecv error: recv got EOF from 127.0.0.1:45154 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260811 02:42:01.759830 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.756125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.2.251.61:41565)
0811 02:42:01.756174 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.756176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.756477 (+ 301us) server_negotiation.cc:407] Connection header received
0811 02:42:01.756583 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.756584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.756596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.756625 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.756839 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.757550 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.758974 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.759275 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.759323 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.759538 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.759577 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.759604 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.759705 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.759730 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:01.763576 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.762705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.2.251.60:37445)
0811 02:42:01.762771 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.762773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.762958 (+ 185us) server_negotiation.cc:407] Connection header received
0811 02:42:01.763057 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.763058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.763070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.763098 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.763502 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45164: BlockingRecv error: recv got EOF from 127.0.0.1:45164 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:42:01.772588 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.769062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.2.251.61:41565)
0811 02:42:01.769112 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.769114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.769411 (+ 297us) server_negotiation.cc:407] Connection header received
0811 02:42:01.769521 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.769522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.769534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.769564 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.769827 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.770550 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.771736 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.772038 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.772088 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.772312 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.772349 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.772376 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.772477 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.772500 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:01.776285 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.775393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.2.251.60:37445)
0811 02:42:01.775440 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.775442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.775664 (+ 222us) server_negotiation.cc:407] Connection header received
0811 02:42:01.775780 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.775782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.775794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.775824 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.776220 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45180: BlockingRecv error: recv got EOF from 127.0.0.1:45180 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:42:01.789207 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.785772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.2.251.61:41565)
0811 02:42:01.785821 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.785823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.786128 (+ 305us) server_negotiation.cc:407] Connection header received
0811 02:42:01.786237 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.786238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.786249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.786279 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.786473 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.787187 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.788374 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.788668 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.788710 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.788930 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.788969 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.788995 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.789092 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.789117 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 02:42:01.792824 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.791935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.2.251.60:37445)
0811 02:42:01.791981 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.791983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.792190 (+ 207us) server_negotiation.cc:407] Connection header received
0811 02:42:01.792286 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.792287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.792300 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.792330 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.792759 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45186: BlockingRecv error: recv got EOF from 127.0.0.1:45186 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:42:01.813912 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.810335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.2.251.61:41565)
0811 02:42:01.810386 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.810389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.810798 (+ 409us) server_negotiation.cc:407] Connection header received
0811 02:42:01.810920 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.810921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.810933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.810962 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.811164 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.811945 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.813054 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.813343 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.813386 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.813586 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.813623 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.813648 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.813801 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.813824 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:01.817463 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.816633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.2.251.60:37445)
0811 02:42:01.816678 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.816680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.816912 (+ 232us) server_negotiation.cc:407] Connection header received
0811 02:42:01.817001 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.817002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.817013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.817039 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.817401 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45190: BlockingRecv error: recv got EOF from 127.0.0.1:45190 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:42:01.852975 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.850777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.2.251.61:41565)
0811 02:42:01.850830 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.850831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.850920 (+ 89us) server_negotiation.cc:407] Connection header received
0811 02:42:01.850989 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.850990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.851003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.851050 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.851165 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.851643 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.852379 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.852568 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.852623 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.852749 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.852782 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.852821 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.852872 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.852892 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260811 02:42:01.855356 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.854697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.2.251.60:37445)
0811 02:42:01.854746 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.854749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.854904 (+ 155us) server_negotiation.cc:407] Connection header received
0811 02:42:01.854946 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.854947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.854958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.855004 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.855302 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45204: BlockingRecv error: recv got EOF from 127.0.0.1:45204 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260811 02:42:01.922771 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:01.920457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.2.251.61:41565)
0811 02:42:01.920515 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.920517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.920569 (+ 52us) server_negotiation.cc:407] Connection header received
0811 02:42:01.920648 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.920649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.920661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.920709 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:01.920839 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.921334 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.922140 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:01.922344 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:01.922402 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:01.922531 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:01.922567 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:01.922607 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:01.922663 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 02:42:01.922684 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260811 02:42:01.925355 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:01.924755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.2.251.60:37445)
0811 02:42:01.924812 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:01.924814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:01.924850 (+ 36us) server_negotiation.cc:407] Connection header received
0811 02:42:01.924918 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:01.924919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:01.924931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:01.924979 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:01.925297 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45216: BlockingRecv error: recv got EOF from 127.0.0.1:45216 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:42:02.056990 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.054804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.2.251.61:41565)
0811 02:42:02.054909 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.054912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.054922 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:02.054951 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.054952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.054963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.055008 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.055136 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.055626 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.056377 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.056562 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.056617 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.056754 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.056785 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.056825 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.056885 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.056905 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260811 02:42:02.059504 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:02.058910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.2.251.60:37445)
0811 02:42:02.058974 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.058976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.059009 (+ 33us) server_negotiation.cc:407] Connection header received
0811 02:42:02.059078 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.059078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.059088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.059123 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:02.059444 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45224: BlockingRecv error: recv got EOF from 127.0.0.1:45224 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 02:42:02.252396 8469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79319316a598423887b95a8be62c94b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79319316a598423887b95a8be62c94b4" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 40815 } } peers { permanent_uuid: "4ce15c84e41c4c8991cbe0978520a1ea" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 41565 } } peers { permanent_uuid: "f2278344e3294fdbbdad3260c9f1f9cb" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 37445 } } }
I20260811 02:42:02.252530 8469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79319316a598423887b95a8be62c94b4 [sys.catalog]: This master's current role is: LEADER
I20260811 02:42:02.319417 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.316858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.2.251.61:41565)
0811 02:42:02.316924 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.316926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.316975 (+ 49us) server_negotiation.cc:407] Connection header received
0811 02:42:02.317041 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.317042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.317051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.317115 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.317252 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.317750 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.318608 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.318815 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.318878 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.319056 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.319091 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.319144 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.319309 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.319334 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260811 02:42:02.321756 8209 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:02.321176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.2.251.60:37445)
0811 02:42:02.321242 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.321244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.321252 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:02.321277 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.321278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.321288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.321332 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:02.321662 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45236: BlockingRecv error: recv got EOF from 127.0.0.1:45236 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 02:42:02.648554 8256 catalog_manager.cc:1259] Loaded cluster ID: 443802e4f426498d818956ee1621e110
I20260811 02:42:02.648666 8256 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: loading cluster ID for follower catalog manager: success
I20260811 02:42:02.649911 8256 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: acquiring CA information for follower catalog manager: success
I20260811 02:42:02.650244 8256 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2278344e3294fdbbdad3260c9f1f9cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:42:02.837185 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.834948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.2.251.61:41565)
0811 02:42:02.835075 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.835077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.835087 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:02.835118 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.835119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.835131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.835185 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.835303 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.835806 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.836544 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.836740 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.836795 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.836922 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.836957 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.836999 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.837064 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.837088 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260811 02:42:02.841355 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.839062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.2.251.60:37445)
0811 02:42:02.839178 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.839180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.839188 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:02.839213 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.839214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.839263 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.839335 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.839480 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.839952 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.840619 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.840807 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.840857 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.841024 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.841150 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.841203 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.841268 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.841287 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260811 02:42:02.846522 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.843027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.2.251.62:40815)
0811 02:42:02.843277 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.843279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.843435 (+ 156us) server_negotiation.cc:407] Connection header received
0811 02:42:02.843513 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.843514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.843565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.843635 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.843738 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.844518 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.845450 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.845764 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.845824 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.846112 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.846221 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.846305 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.846407 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.846434 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":55,"threads_started":1}
I20260811 02:42:02.846659 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.843020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.2.251.61:41565)
0811 02:42:02.843067 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.843069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.843312 (+ 243us) server_negotiation.cc:407] Connection header received
0811 02:42:02.843499 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.843501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.843514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.843540 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.843696 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.844219 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.845137 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.845566 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.845684 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.845993 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.846115 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.846147 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.846309 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.846550 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:42:02.847532 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:02.843137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.2.251.60:37445)
0811 02:42:02.843184 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:02.843186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:02.843282 (+ 96us) server_negotiation.cc:407] Connection header received
0811 02:42:02.843342 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:02.843343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:02.843421 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:02.843486 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:02.843795 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.844517 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.846492 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:02.846793 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:02.846852 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:02.847107 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:02.847141 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:02.847183 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:02.847463 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 02:42:02.847483 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:42:02.851900 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7849
I20260811 02:42:02.869139 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 7970
I20260811 02:42:02.881683 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8093
2026-08-11T02:42:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3091 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 02:42:02 dist-test-slave-cvwc krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 02:42:02 dist-test-slave-cvwc krb5kdc[8529](info): set up 2 sockets
Aug 11 02:42:02 dist-test-slave-cvwc krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260811 02:42:04.112068 3052 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/test-user.keytab.
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T02:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T02:42:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.251.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
Entry for principal kudu/127.2.251.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab.
I20260811 02:42:04.249436 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34509
--webserver_interface=127.2.251.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34509,127.2.251.61:45281,127.2.251.60:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 02:42:04.340307 8549 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 02:42:04.340566 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:42:04.340585 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:42:04.340607 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:42:04.341837 8549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:42:04.341876 8549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:42:04.341889 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:42:04.341902 8549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:42:04.341912 8549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:42:04.343474 8549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34509,127.2.251.61:45281,127.2.251.60:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.62.keytab
--principal=kudu/127.2.251.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.62:34509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.251.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:42:04.343698 8549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:42:04.343896 8549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:42:04.349025 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:04.349015 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:04.349095 8549 server_base.cc:1034] running on GCE node
W20260811 02:42:04.349069 8563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:04.349499 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:42:04.349813 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:42:04.350982 8549 hybrid_clock.cc:648] HybridClock initialized: now 1786416124350970 us; error 33 us; skew 500 ppm
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:42:04.353704 8549 init.cc:377] Logged in from keytab as kudu/127.2.251.62@KRBTEST.COM (short username kudu)
I20260811 02:42:04.354216 8549 webserver.cc:460] Webserver started at http://127.2.251.62:46717/ using document root <none> and password file <none>
I20260811 02:42:04.354470 8549 fs_manager.cc:359] Metadata directory not provided
I20260811 02:42:04.354521 8549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:42:04.354650 8549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:42:04.355800 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data/instance:
uuid: "952e168844564fd588c259a1101b1d0d"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "8551dafa1ac2c504e42e363125fdb76c"
server_key_iv: "9053be7e9a59e97ef2d26b6de0a1ac44"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.356148 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal/instance:
uuid: "952e168844564fd588c259a1101b1d0d"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "8551dafa1ac2c504e42e363125fdb76c"
server_key_iv: "9053be7e9a59e97ef2d26b6de0a1ac44"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.357513 8549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260811 02:42:04.358428 8572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:42:04.358675 8549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 02:42:04.358783 8549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal
uuid: "952e168844564fd588c259a1101b1d0d"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "8551dafa1ac2c504e42e363125fdb76c"
server_key_iv: "9053be7e9a59e97ef2d26b6de0a1ac44"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.358863 8549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:42:04.373371 8549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:42:04.375929 8549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:42:04.376091 8549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:42:04.380178 8549 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.62:34509
I20260811 02:42:04.380234 8663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.62:34509 every 8 connection(s)
I20260811 02:42:04.380594 8549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/data/info.pb
I20260811 02:42:04.381624 8665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:42:04.383324 8665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:04.387542 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8549
I20260811 02:42:04.387655 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-0/wal/instance
I20260811 02:42:04.387873 3052 external_mini_cluster.cc:1428] Setting key af7bf0d030e8ef2ece041c1b0fd79d46
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:42:04.395749 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.384187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.2.251.62:34509)
0811 02:42:04.384316 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.384320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.384399 (+ 79us) server_negotiation.cc:407] Connection header received
0811 02:42:04.385024 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.385026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.385044 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.385133 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.386183 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.387034 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.388462 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.388903 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.389022 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.392210 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.392218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.392219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.392226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.393829 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.394393 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.394396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.394396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.394491 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.394774 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.394775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.394775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.395051 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.395126 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.395407 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.395594 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
I20260811 02:42:04.395756 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.384020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34509 (local address 127.0.0.1:52074)
0811 02:42:04.384210 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.384259 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.384845 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.385255 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.385259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.385638 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.385969 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.385975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.387141 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.387142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.388368 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.388372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.389052 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.389053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.389510 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.390147 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.390157 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.391904 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.393914 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.393924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.393927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.394293 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.394559 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.394560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.394560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.394683 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.395128 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.395136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.395219 (+ 83us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.395335 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.395535 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260811 02:42:04.396519 8665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:04.396889 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.396743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53230)
0811 02:42:04.396797 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.396861 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 02:42:04.397037 8576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.251.61@KRBTEST.COM; defaulting to no policy
I20260811 02:42:04.397464 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.397369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53240)
0811 02:42:04.397412 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.397441 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:42:04.397574 8665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:45281: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab.
Entry for principal kudu/127.2.251.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab.
I20260811 02:42:04.421181 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:45281
--webserver_interface=127.2.251.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab
--principal=kudu/127.2.251.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34509,127.2.251.61:45281,127.2.251.60:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:42:04.446805 8665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } attempt: 1
I20260811 02:42:04.447374 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.447191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53250)
0811 02:42:04.447267 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.447323 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 02:42:04.447745 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.447664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53264)
0811 02:42:04.447698 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.447727 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260811 02:42:04.447903 8665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:45281: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
I20260811 02:42:04.516124 8665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } attempt: 2
I20260811 02:42:04.516924 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.516659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53278)
0811 02:42:04.516798 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.516863 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260811 02:42:04.516961 8679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 02:42:04.517252 8679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:42:04.517292 8679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:42:04.517330 8679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 02:42:04.517361 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.517259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53282)
0811 02:42:04.517307 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.517335 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 02:42:04.517517 8665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.61:45281: Network error: Client connection negotiation failed: client connection to 127.2.251.61:45281: connect: Connection refused (error 111)
W20260811 02:42:04.518816 8679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:42:04.518888 8679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:42:04.518904 8679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:42:04.518916 8679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:42:04.518927 8679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:42:04.520905 8679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34509,127.2.251.61:45281,127.2.251.60:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.61.keytab
--principal=kudu/127.2.251.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.61:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.251.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:42:04.521203 8679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:42:04.521451 8679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 02:42:04.527086 8685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:04.527009 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:04.527146 8692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:04.527405 8679 server_base.cc:1034] running on GCE node
I20260811 02:42:04.527833 8679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:42:04.528358 8679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:42:04.529562 8679 hybrid_clock.cc:648] HybridClock initialized: now 1786416124529515 us; error 81 us; skew 500 ppm
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:42:04.532853 8679 init.cc:377] Logged in from keytab as kudu/127.2.251.61@KRBTEST.COM (short username kudu)
I20260811 02:42:04.533483 8679 webserver.cc:460] Webserver started at http://127.2.251.61:40523/ using document root <none> and password file <none>
I20260811 02:42:04.533841 8679 fs_manager.cc:359] Metadata directory not provided
I20260811 02:42:04.533900 8679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:42:04.534161 8679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:42:04.535444 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data/instance:
uuid: "8da92642372c4d77b31fccac9bce9791"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "87758f117acc9e0358745cdc648201d5"
server_key_iv: "ec5e7c7346ae5a315f3e407f6389b22f"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.535813 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal/instance:
uuid: "8da92642372c4d77b31fccac9bce9791"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "87758f117acc9e0358745cdc648201d5"
server_key_iv: "ec5e7c7346ae5a315f3e407f6389b22f"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.537338 8679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:42:04.538214 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:42:04.538376 8679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 02:42:04.538483 8679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal
uuid: "8da92642372c4d77b31fccac9bce9791"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "87758f117acc9e0358745cdc648201d5"
server_key_iv: "ec5e7c7346ae5a315f3e407f6389b22f"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.538553 8679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:42:04.553324 8679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:42:04.555859 8679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:42:04.556037 8679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:42:04.560732 8679 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.61:45281
I20260811 02:42:04.560755 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.61:45281 every 8 connection(s)
I20260811 02:42:04.561235 8679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/data/info.pb
I20260811 02:42:04.562768 8794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:42:04.564998 8794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:04.567953 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8679
I20260811 02:42:04.568081 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-1/wal/instance
I20260811 02:42:04.568414 3052 external_mini_cluster.cc:1428] Setting key ad5fa53b50e6b429725e76f64ea82bff
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:42:04.577926 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.565913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.2.251.62:34509)
0811 02:42:04.565968 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.565970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.566396 (+ 426us) server_negotiation.cc:407] Connection header received
0811 02:42:04.566947 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.566949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.566964 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.567027 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.567897 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.568712 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.570254 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.570765 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.570920 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.574087 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.574090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.574091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.574102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.575930 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.576683 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.576684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.576685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.576761 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.577109 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.577110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.577110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.577255 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.577336 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.577744 (+ 408us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.577848 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260811 02:42:04.578204 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.565915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34509 (local address 127.0.0.1:52076)
0811 02:42:04.566155 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.566199 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.566735 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.567213 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.567217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.567531 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.567770 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.567775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.568929 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.568931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.570150 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.570156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.570994 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.570996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.571205 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.571857 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.571869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.573886 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.576093 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.576114 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.576118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.576560 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.576839 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.576840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.576840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.577014 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.577338 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.577345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.577564 (+ 219us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.577693 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.577932 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260811 02:42:04.578923 8794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.251.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.251.60@KRBTEST.COM" created.
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:42:04.587014 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.579208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53288)
0811 02:42:04.579256 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.579267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.579308 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.579640 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.579641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.579773 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.579838 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.579840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.580707 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.580708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.581240 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.581242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.581735 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.581737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.581805 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.582314 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.582319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.583656 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.585853 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.585854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.585855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.586024 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.586271 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.586272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.586273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.586340 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.586695 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.586697 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.586745 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.586808 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.586927 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 02:42:04.587134 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.579278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.2.251.61:45281)
0811 02:42:04.579477 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.579483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.579509 (+ 26us) server_negotiation.cc:407] Connection header received
0811 02:42:04.579534 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.579536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.579552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.579635 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.579910 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.580628 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.581319 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.581653 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.581737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.583755 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.583767 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.583767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.583775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.585703 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.586108 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.586112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.586113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.586182 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.586407 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.586408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.586408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.586609 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.586694 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.586856 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.586997 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260811 02:42:04.587916 8794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:04.588335 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.588206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:53984)
0811 02:42:04.588262 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.588311 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 02:42:04.588472 8710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
I20260811 02:42:04.588889 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.588805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:53994)
0811 02:42:04.588842 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.588867 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260811 02:42:04.588990 8794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41245: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.251.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab.
Entry for principal kudu/127.2.251.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab.
I20260811 02:42:04.602527 3052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRZ17kF/build/release/bin/kudu
/tmp/dist-test-taskRZ17kF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:41245
--webserver_interface=127.2.251.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab
--principal=kudu/127.2.251.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.251.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.251.62:34509,127.2.251.61:45281,127.2.251.60:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 02:42:04.608677 8665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } attempt: 3
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:42:04.617085 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.609139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53292)
0811 02:42:04.609213 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.609224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.609271 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.609501 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.609502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.609702 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.609827 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.609830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.610761 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.610764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.611409 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.611413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.611849 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.611851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.611936 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.612470 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.612476 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.614297 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.616001 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.616003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.616004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.616213 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.616410 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.616411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.616411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.616474 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.616712 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.616714 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.616775 (+ 61us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.616845 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.616976 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 02:42:04.617085 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.609287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.2.251.61:45281)
0811 02:42:04.609383 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.609385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.609393 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:04.609418 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.609419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.609432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.609479 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.609909 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.610582 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.611523 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.611772 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.611839 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.614441 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.614443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.614443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.614454 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.615884 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.616292 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.616293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.616293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.616351 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.616526 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.616527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.616527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.616653 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.616712 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.616883 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.616977 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 02:42:04.617668 8665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:04.618057 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.617952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54002)
0811 02:42:04.617998 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.618028 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 02:42:04.618379 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.618317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54014)
0811 02:42:04.618344 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.618366 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260811 02:42:04.618453 8665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41245: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
I20260811 02:42:04.638278 8794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } attempt: 1
I20260811 02:42:04.638829 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.638706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54028)
0811 02:42:04.638763 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.638799 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 02:42:04.639158 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.639078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54038)
0811 02:42:04.639114 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.639138 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260811 02:42:04.639251 8794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41245: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
I20260811 02:42:04.649638 8665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } attempt: 1
I20260811 02:42:04.650202 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.650071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54046)
0811 02:42:04.650127 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.650168 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 02:42:04.651070 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.650904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54058)
0811 02:42:04.650981 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.651040 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 02:42:04.651330 8665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41245: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
W20260811 02:42:04.703688 8807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 02:42:04.703960 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 02:42:04.703979 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 02:42:04.704002 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 02:42:04.705178 8807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 02:42:04.705235 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 02:42:04.705247 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 02:42:04.705260 8807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 02:42:04.705272 8807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 02:42:04.706919 8807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.251.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.251.62:34509,127.2.251.61:45281,127.2.251.60:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/krb5kdc/kudu_127.2.251.60.keytab
--principal=kudu/127.2.251.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.251.60:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.251.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 01:46:40 UTC on ccycloud.quasar-juvatc.root.comops.site
build id 4579571
I20260811 02:42:04.707396 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 02:42:04.707418 8794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } attempt: 2
I20260811 02:42:04.707819 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 02:42:04.707864 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.707737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54060)
0811 02:42:04.707802 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.707837 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 02:42:04.708379 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.708210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54062)
0811 02:42:04.708276 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.708338 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 02:42:04.708535 8794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41245: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
W20260811 02:42:04.713939 8818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 02:42:04.713929 8822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:04.714201 8807 server_base.cc:1034] running on GCE node
W20260811 02:42:04.713929 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 02:42:04.714532 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 02:42:04.714895 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 02:42:04.716125 8807 hybrid_clock.cc:648] HybridClock initialized: now 1786416124716092 us; error 36 us; skew 500 ppm
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 02:42:04.722892 8807 init.cc:377] Logged in from keytab as kudu/127.2.251.60@KRBTEST.COM (short username kudu)
I20260811 02:42:04.723555 8807 webserver.cc:460] Webserver started at http://127.2.251.60:37785/ using document root <none> and password file <none>
I20260811 02:42:04.723796 8807 fs_manager.cc:359] Metadata directory not provided
I20260811 02:42:04.723863 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 02:42:04.724011 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 02:42:04.725152 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data/instance:
uuid: "95d07aaa3e0e4c96aa0792db24a7655b"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "a61485395773426692a92339b301eded"
server_key_iv: "6aca5123b732540de634ed46ca680d88"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.725515 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal/instance:
uuid: "95d07aaa3e0e4c96aa0792db24a7655b"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "a61485395773426692a92339b301eded"
server_key_iv: "6aca5123b732540de634ed46ca680d88"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.726569 8665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } attempt: 2
I20260811 02:42:04.727067 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.726901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54074)
0811 02:42:04.726979 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.727029 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 02:42:04.727100 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260811 02:42:04.727523 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.727411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54082)
0811 02:42:04.727462 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.727497 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260811 02:42:04.727711 8665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.251.60:41245: Network error: Client connection negotiation failed: client connection to 127.2.251.60:41245: connect: Connection refused (error 111)
I20260811 02:42:04.728211 8838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 02:42:04.728503 8807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 02:42:04.728606 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data,/tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal
uuid: "95d07aaa3e0e4c96aa0792db24a7655b"
format_stamp: "Formatted at 2026-08-11 02:42:04 on dist-test-slave-cvwc"
server_key: "a61485395773426692a92339b301eded"
server_key_iv: "6aca5123b732540de634ed46ca680d88"
server_key_version: "encryptionkey@0"
I20260811 02:42:04.728684 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 02:42:04.761464 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 02:42:04.764397 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 02:42:04.764568 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 02:42:04.768970 8807 rpc_server.cc:307] RPC server started. Bound to: 127.2.251.60:41245
I20260811 02:42:04.769009 8918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.251.60:41245 every 8 connection(s)
I20260811 02:42:04.769526 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/data/info.pb
I20260811 02:42:04.770123 3052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRZ17kF/build/release/bin/kudu as pid 8807
I20260811 02:42:04.770215 3052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRZ17kF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786415831315317-3052-0/minicluster-data/master-2/wal/instance
I20260811 02:42:04.770413 3052 external_mini_cluster.cc:1428] Setting key 8c3eaf137d59684cb8830913992bc7c7
I20260811 02:42:04.770399 8920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 02:42:04.772913 8920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:42:04.780745 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.773646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.2.251.62:34509)
0811 02:42:04.773702 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.773704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.773877 (+ 173us) server_negotiation.cc:407] Connection header received
0811 02:42:04.773948 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.773949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.773960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.774007 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.774449 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.774911 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.775647 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.775834 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.775889 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.778151 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.778153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.778153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.778161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.779574 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.779910 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.779911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.779912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.779967 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.780163 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.780164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.780165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.780281 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.780337 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.780561 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.780646 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:42:04.780928 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.773549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.62:34509 (local address 127.0.0.1:52082)
0811 02:42:04.773759 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.773810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.773885 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.774025 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.774027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.774305 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.774371 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.774373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.774993 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.774994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.775567 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.775568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.775925 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.775926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.775983 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.776434 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.776442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.778058 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.779641 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.779647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.779649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.779836 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.780023 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.780024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.780024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.780108 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.780337 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.780344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.780451 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.780532 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.780684 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260811 02:42:04.781064 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.770969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.2.251.60:41245)
0811 02:42:04.771140 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.771149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.771178 (+ 29us) server_negotiation.cc:407] Connection header received
0811 02:42:04.771601 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.771608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.771655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.771774 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.772615 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.773901 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.774661 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.775077 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.775172 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.777567 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.777574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.777575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.777582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.779498 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.779921 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.779925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.779926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.780045 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.780232 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.780233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.780233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.780463 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.780709 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.780793 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.780878 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260811 02:42:04.781539 8920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.62@KRBTEST.COM
I20260811 02:42:04.788546 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.781829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.61:45281 (local address 127.0.0.1:53300)
0811 02:42:04.781877 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.781888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.781926 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.782076 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.782077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.782212 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.782276 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.782277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.782907 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.782908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.783411 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.783412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.783747 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.783748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.783817 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.784232 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.784236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.785610 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.787243 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.787244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.787245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.787698 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.787921 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.787922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.787922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.787987 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.788218 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.788219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.788266 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.788327 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.788439 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 02:42:04.788546 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.781907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.2.251.61:45281)
0811 02:42:04.781953 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.781954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.781963 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:42:04.781988 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.781989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.781999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.782056 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.782345 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.782840 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.783481 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.783686 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.783752 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.785704 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.785705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.785706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.785739 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.787099 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.787790 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.787791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.787792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.787845 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.788040 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.788041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.788041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.788160 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.788216 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.788360 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.788439 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 02:42:04.788947 8920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } has no permanent_uuid. Determining permanent_uuid...
I20260811 02:42:04.790913 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.782041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.2.251.62:34509)
0811 02:42:04.782076 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.782078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.782085 (+ 7us) server_negotiation.cc:407] Connection header received
0811 02:42:04.782131 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.782131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.782140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.782183 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.782616 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.783069 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.783717 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.783915 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.783968 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.786201 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.786203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.786203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.786217 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.789397 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.789728 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.789728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.789729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.789791 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.789985 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.789985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.789986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.790116 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.790528 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.790713 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.790807 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.60@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.251.61@KRBTEST.COM
I20260811 02:42:04.797444 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.789193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54092)
0811 02:42:04.789233 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.789244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.789282 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.789779 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.789780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.789925 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.789974 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.789976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.791347 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.791348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.792064 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.792067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.792557 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.792562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.792697 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.793146 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.793152 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.794842 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.796526 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.796527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.796528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.796688 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.796859 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.796860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.796860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.796917 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.797149 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.797150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.797195 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.797257 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.797340 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260811 02:42:04.797612 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.789294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.2.251.60:41245)
0811 02:42:04.789344 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.789346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.789354 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:04.789376 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.789377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.789388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.789456 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.790046 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.791226 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.792140 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.792462 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.792534 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.794927 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.794928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.794929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.794935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.796450 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.796751 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.796752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.796752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.796806 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.796967 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.796968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.796968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.797088 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.797153 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.797291 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.797417 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":29}
I20260811 02:42:04.799445 8920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: Bootstrap starting.
I20260811 02:42:04.799667 8794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } attempt: 3
I20260811 02:42:04.800408 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.791821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.2.251.61:45281)
0811 02:42:04.791878 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.791880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.791888 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:04.791911 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.791912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.791923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.792004 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.792657 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.793370 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.794315 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.794621 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.794701 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.797128 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.797129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.797130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.797138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.798807 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.799232 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.799234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.799234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.799302 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.799550 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.799551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.799551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.799698 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.799864 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.800218 (+ 354us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.800317 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 02:42:04.800724 8920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: Neither blocks nor log segments found. Creating new log.
I20260811 02:42:04.801071 8920 log.cc:824] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: Log is configured to *not* fsync() on all Append() calls
I20260811 02:42:04.802116 3052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 02:42:04.802441 8920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: No bootstrap required, opened a new log
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.61@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:42:04.803519 8618 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "952e168844564fd588c259a1101b1d0d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52086
I20260811 02:42:04.805837 8920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.806030 8920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d07aaa3e0e4c96aa0792db24a7655b, State: Initialized, Role: FOLLOWER
I20260811 02:42:04.806146 8920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.806478 8931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.806555 8920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:42:04.806571 8931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:04.807273 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.799930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54100)
0811 02:42:04.799984 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.799994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.800025 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.800260 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.800261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.800417 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.800465 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.800467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.801307 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.801308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.801976 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.801979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.802389 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.802390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.802459 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.802962 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.802967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.804276 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.805826 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.805827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.805828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.806021 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.806304 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.806305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.806305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.806368 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.806924 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.806926 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.806980 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.807055 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.807172 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 02:42:04.807286 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.799979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.2.251.60:41245)
0811 02:42:04.800026 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.800029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.800062 (+ 33us) server_negotiation.cc:407] Connection header received
0811 02:42:04.800086 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.800087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.800097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.800145 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.800555 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.801233 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.802051 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.802312 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.802377 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.804359 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.804361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.804361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.804370 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.805680 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.806179 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.806180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.806180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.806235 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.806455 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.806456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.806456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.806839 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.806925 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.807101 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.807198 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260811 02:42:04.809144 8794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: Bootstrap starting.
I20260811 02:42:04.809966 8794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: Neither blocks nor log segments found. Creating new log.
W20260811 02:42:04.808485 8951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 02:42:04.810283 8794 log.cc:824] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: Log is configured to *not* fsync() on all Append() calls
W20260811 02:42:04.810401 8951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:42:04.811179 8794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: No bootstrap required, opened a new log
I20260811 02:42:04.813230 8794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.813375 8794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da92642372c4d77b31fccac9bce9791, State: Initialized, Role: FOLLOWER
I20260811 02:42:04.813449 8794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.813833 8794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:42:04.810858 8807 master_runner.cc:186] Error getting master registration for 127.2.251.62:34509: OK, instance_id { permanent_uuid: "952e168844564fd588c259a1101b1d0d" instance_seqno: 1786416124379023 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 02:42:04.813946 8807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:42:04.813877 8959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.814029 8959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 02:42:04.815742 8974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:42:04.815856 8974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:42:04.815896 8679 master_runner.cc:186] Error getting master registration for 127.2.251.62:34509: OK, instance_id { permanent_uuid: "952e168844564fd588c259a1101b1d0d" instance_seqno: 1786416124379023 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 02:42:04.818907 8679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:42:04.826963 8665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } attempt: 3
Aug 11 02:42:04 dist-test-slave-cvwc krb5kdc[8529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786416124, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.251.62@KRBTEST.COM for kudu/127.2.251.60@KRBTEST.COM
I20260811 02:42:04.834851 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.827518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.2.251.60:41245)
0811 02:42:04.827560 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.827562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.827570 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:04.827599 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.827600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.827610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.827657 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.827977 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.828743 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.829417 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.829773 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.829833 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.832008 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.832010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.832010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.832020 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.833682 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.834047 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.834048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.834048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.834109 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.834289 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.834290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.834290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.834455 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.834513 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.834661 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.834742 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:42:04.834851 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.827373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.251.60:41245 (local address 127.0.0.1:54108)
0811 02:42:04.827423 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0811 02:42:04.827437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 02:42:04.827488 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 02:42:04.827671 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 02:42:04.827673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 02:42:04.827848 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 02:42:04.827895 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.827897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.828816 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.828818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.829331 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 02:42:04.829333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.829896 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.829897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 02:42:04.829956 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.830413 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 02:42:04.830419 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 02:42:04.831907 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 02:42:04.833798 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.833799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.833800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.833979 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.834157 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 02:42:04.834158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 02:42:04.834158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0811 02:42:04.834235 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 02:42:04.834513 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 02:42:04.834515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 02:42:04.834567 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 02:42:04.834627 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 02:42:04.834742 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260811 02:42:04.837078 8665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d: Bootstrap starting.
I20260811 02:42:04.837906 8665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d: Neither blocks nor log segments found. Creating new log.
I20260811 02:42:04.838217 8665 log.cc:824] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d: Log is configured to *not* fsync() on all Append() calls
I20260811 02:42:04.839134 8665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d: No bootstrap required, opened a new log
I20260811 02:42:04.840231 8665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.840377 8665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952e168844564fd588c259a1101b1d0d, State: Initialized, Role: FOLLOWER
I20260811 02:42:04.840461 8665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.840775 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.840895 8618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 02:42:04.840843 8665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 02:42:04.840919 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:04.840955 8618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:42:04.842769 8618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
W20260811 02:42:04.843636 9003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 02:42:04.843732 9003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 02:42:04.845587 8745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952e168844564fd588c259a1101b1d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8da92642372c4d77b31fccac9bce9791"
I20260811 02:42:04.845813 8745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:42:04.846843 8745 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952e168844564fd588c259a1101b1d0d in term 1.
I20260811 02:42:04.847050 8576 leader_election.cc:302] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da92642372c4d77b31fccac9bce9791, 952e168844564fd588c259a1101b1d0d; no voters:
I20260811 02:42:04.847182 8982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 02:42:04.847369 8982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [term 1 LEADER]: Becoming Leader. State: Replica: 952e168844564fd588c259a1101b1d0d, State: Running, Role: LEADER
I20260811 02:42:04.847466 8982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } }
I20260811 02:42:04.847942 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952e168844564fd588c259a1101b1d0d. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.848011 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: This master's current role is: LEADER
I20260811 02:42:04.848455 8878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952e168844564fd588c259a1101b1d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95d07aaa3e0e4c96aa0792db24a7655b"
I20260811 02:42:04.848599 8878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [term 0 FOLLOWER]: Advancing to term 1
I20260811 02:42:04.849581 8878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952e168844564fd588c259a1101b1d0d in term 1.
I20260811 02:42:04.849841 8618 leader_election.cc:288] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [CANDIDATE]: Term 1 election: Requested vote from peers 8da92642372c4d77b31fccac9bce9791 (127.2.251.61:45281), 95d07aaa3e0e4c96aa0792db24a7655b (127.2.251.60:41245)
I20260811 02:42:04.851694 8549 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260811 02:42:04.851804 8549 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 02:42:04.854444 9008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 02:42:04.854673 9008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 02:42:04.856562 8878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [term 1 FOLLOWER]: Refusing update from remote peer 952e168844564fd588c259a1101b1d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:42:04.857036 8745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [term 1 FOLLOWER]: Refusing update from remote peer 952e168844564fd588c259a1101b1d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 02:42:04.857443 8982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:42:04.857623 8982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 02:42:04.859452 8959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952e168844564fd588c259a1101b1d0d. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.859571 8959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:04.859521 8931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 952e168844564fd588c259a1101b1d0d. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.859624 8931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:04.870492 9015 mvcc.cc:205] Tried to move back new op lower bound from 7317160447409254400 to 7317160447376019456. Current Snapshot: MvccSnapshot[applied={T|T < 7317160447409254400}]
I20260811 02:42:04.871037 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.871148 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: This master's current role is: LEADER
I20260811 02:42:04.871188 8982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.871224 8982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d [sys.catalog]: This master's current role is: LEADER
I20260811 02:42:04.871659 8959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.871770 8959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:04.872006 9020 mvcc.cc:205] Tried to move back new op lower bound from 7317160447409254400 to 7317160447376019456. Current Snapshot: MvccSnapshot[applied={T|T < 7317160447409254400}]
I20260811 02:42:04.872900 8931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "952e168844564fd588c259a1101b1d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952e168844564fd588c259a1101b1d0d" member_type: VOTER last_known_addr { host: "127.2.251.62" port: 34509 } } peers { permanent_uuid: "8da92642372c4d77b31fccac9bce9791" member_type: VOTER last_known_addr { host: "127.2.251.61" port: 45281 } } peers { permanent_uuid: "95d07aaa3e0e4c96aa0792db24a7655b" member_type: VOTER last_known_addr { host: "127.2.251.60" port: 41245 } } }
I20260811 02:42:04.873010 8931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 02:42:04.874738 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.858044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.2.251.61:45281)
0811 02:42:04.858092 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.858094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.862719 (+ 4625us) server_negotiation.cc:407] Connection header received
0811 02:42:04.862863 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.862865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.862877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.862907 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.863811 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.864566 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.865653 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.865996 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.866038 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.867638 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.867640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.867640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.867647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.869928 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.873851 (+ 3923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.873853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.873853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.873924 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.874129 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.874130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.874130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.874266 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.874341 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.874564 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.874647 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:42:04.877121 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.859672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.2.251.60:41245)
0811 02:42:04.859719 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.859721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.862706 (+ 2985us) server_negotiation.cc:407] Connection header received
0811 02:42:04.865889 (+ 3183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.865891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.865903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.865933 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.867454 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.868174 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.869199 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.869513 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.869553 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.873951 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.873953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.873953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.873962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.875842 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.876234 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.876235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.876235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.876311 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.876527 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.876528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.876528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.876663 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.876738 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.876952 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.877033 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260811 02:42:04.880594 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:04.862410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.2.251.62:34509)
0811 02:42:04.862470 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.862472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.866811 (+ 4339us) server_negotiation.cc:407] Connection header received
0811 02:42:04.866909 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.866910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.866922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.866951 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.868443 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.869160 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.875586 (+ 6426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:04.875919 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:04.875962 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:04.877659 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 02:42:04.877661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 02:42:04.877661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 02:42:04.877670 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 02:42:04.879312 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.879720 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.879721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.879721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.879793 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 02:42:04.879992 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 02:42:04.879993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 02:42:04.879993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0811 02:42:04.880126 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 02:42:04.880202 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:04.880404 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 02:42:04.880490 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 02:42:04.881429 9008 catalog_manager.cc:1347] Generated new cluster ID: c09a4d45fb034d81837ace88264251a0
I20260811 02:42:04.881502 9008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 02:42:04.895390 9008 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 02:42:04.896164 9008 catalog_manager.cc:1504] Loading token signing keys...
I20260811 02:42:04.904943 9008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 952e168844564fd588c259a1101b1d0d: Generated new TSK 0
I20260811 02:42:04.905225 9008 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 02:42:04.931289 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.929142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.2.251.61:45281)
0811 02:42:04.929198 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.929200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.930438 (+ 1238us) server_negotiation.cc:407] Connection header received
0811 02:42:04.930567 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.930569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.930583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.930618 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.931206 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53324: BlockingRecv error: recv got EOF from 127.0.0.1:53324 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260811 02:42:04.932341 8926 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.929588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.2.251.60:41245)
0811 02:42:04.929639 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.929640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.931635 (+ 1995us) server_negotiation.cc:407] Connection header received
0811 02:42:04.931811 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.931812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.931827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.931863 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.932263 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54128: BlockingRecv error: recv got EOF from 127.0.0.1:54128 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260811 02:42:04.933324 8670 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.930055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.2.251.62:34509)
0811 02:42:04.930100 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.930102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.932670 (+ 2568us) server_negotiation.cc:407] Connection header received
0811 02:42:04.932803 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.932804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.932819 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.932853 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.933249 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52104: BlockingRecv error: recv got EOF from 127.0.0.1:52104 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260811 02:42:04.939786 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.938662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.2.251.61:45281)
0811 02:42:04.938738 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.938741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.939039 (+ 298us) server_negotiation.cc:407] Connection header received
0811 02:42:04.939172 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.939173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.939196 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.939231 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.939706 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53336: BlockingRecv error: recv got EOF from 127.0.0.1:53336 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260811 02:42:04.943729 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.942618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.2.251.61:45281)
0811 02:42:04.942665 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.942667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.943011 (+ 344us) server_negotiation.cc:407] Connection header received
0811 02:42:04.943151 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.943152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.943165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.943200 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.943648 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53338: BlockingRecv error: recv got EOF from 127.0.0.1:53338 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260811 02:42:04.948102 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.947054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.2.251.61:45281)
0811 02:42:04.947101 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.947103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.947427 (+ 324us) server_negotiation.cc:407] Connection header received
0811 02:42:04.947548 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.947549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.947563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.947599 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.948036 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53350: BlockingRecv error: recv got EOF from 127.0.0.1:53350 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260811 02:42:04.954685 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.953560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.2.251.61:45281)
0811 02:42:04.953610 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.953612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.953998 (+ 386us) server_negotiation.cc:407] Connection header received
0811 02:42:04.954119 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.954120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.954133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.954168 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.954617 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53352: BlockingRecv error: recv got EOF from 127.0.0.1:53352 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:04.965289 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.964201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.2.251.61:45281)
0811 02:42:04.964255 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.964258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.964597 (+ 339us) server_negotiation.cc:407] Connection header received
0811 02:42:04.964726 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.964727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.964741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.964776 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.965219 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53358: BlockingRecv error: recv got EOF from 127.0.0.1:53358 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:04.984180 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:04.982934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.2.251.61:45281)
0811 02:42:04.982985 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:04.982987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:04.983322 (+ 335us) server_negotiation.cc:407] Connection header received
0811 02:42:04.983440 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:04.983441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:04.983454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:04.983592 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:04.984109 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53370: BlockingRecv error: recv got EOF from 127.0.0.1:53370 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260811 02:42:05.018406 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:05.017539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.2.251.61:45281)
0811 02:42:05.017596 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.017597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.017739 (+ 142us) server_negotiation.cc:407] Connection header received
0811 02:42:05.017798 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.017799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.017811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.017862 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:05.018353 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53374: BlockingRecv error: recv got EOF from 127.0.0.1:53374 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260811 02:42:05.084604 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:05.083937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.2.251.61:45281)
0811 02:42:05.084012 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.084014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.084024 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:05.084058 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.084059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.084070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.084118 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:05.084458 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53378: BlockingRecv error: recv got EOF from 127.0.0.1:53378 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 02:42:05.214857 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:05.214067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.2.251.61:45281)
0811 02:42:05.214128 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.214131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.214198 (+ 67us) server_negotiation.cc:407] Connection header received
0811 02:42:05.214254 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.214255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.214266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.214372 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:05.214772 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53390: BlockingRecv error: recv got EOF from 127.0.0.1:53390 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260811 02:42:05.472741 8802 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 02:42:05.472103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.2.251.61:45281)
0811 02:42:05.472201 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.472203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.472213 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:05.472244 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.472245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.472255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.472333 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 02:42:05.472675 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53398: BlockingRecv error: recv got EOF from 127.0.0.1:53398 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 02:42:05.811000 8951 catalog_manager.cc:1259] Loaded cluster ID: c09a4d45fb034d81837ace88264251a0
I20260811 02:42:05.811092 8951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: loading cluster ID for follower catalog manager: success
I20260811 02:42:05.811918 8951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: acquiring CA information for follower catalog manager: success
I20260811 02:42:05.812211 8951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95d07aaa3e0e4c96aa0792db24a7655b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:42:05.816327 8974 catalog_manager.cc:1259] Loaded cluster ID: c09a4d45fb034d81837ace88264251a0
I20260811 02:42:05.816391 8974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: loading cluster ID for follower catalog manager: success
I20260811 02:42:05.817178 8974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: acquiring CA information for follower catalog manager: success
I20260811 02:42:05.817473 8974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8da92642372c4d77b31fccac9bce9791: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 02:42:05.989526 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:05.986005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.2.251.61:45281)
0811 02:42:05.986132 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.986134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.986151 (+ 17us) server_negotiation.cc:407] Connection header received
0811 02:42:05.986259 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.986260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.986329 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.986414 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:05.986637 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.987268 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.988515 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.988708 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.988771 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:05.989023 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:05.989149 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:05.989238 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:05.989391 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 02:42:05.989415 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260811 02:42:05.994822 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:05.991847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.2.251.60:41245)
0811 02:42:05.991979 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.991981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.991991 (+ 10us) server_negotiation.cc:407] Connection header received
0811 02:42:05.992032 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.992033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.992083 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.992161 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:05.992400 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.993014 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.993830 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.994040 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.994101 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:05.994373 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:05.994498 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:05.994568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:05.994717 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 02:42:05.994745 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260811 02:42:05.999960 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:05.997030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.2.251.61:45281)
0811 02:42:05.997077 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.997079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.997256 (+ 177us) server_negotiation.cc:407] Connection header received
0811 02:42:05.997399 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.997401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.997414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.997463 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:05.997630 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.998381 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.999151 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.999455 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.999494 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:05.999698 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:05.999734 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:05.999755 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:05.999842 (+ 87us) server_negotiation.cc:299] Negotiation successful
0811 02:42:05.999872 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260811 02:42:05.999994 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:05.997258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.2.251.60:41245)
0811 02:42:05.997296 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.997308 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.997316 (+ 8us) server_negotiation.cc:407] Connection header received
0811 02:42:05.997346 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.997347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.997355 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.997400 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:05.997571 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.998331 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.999290 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.999489 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.999548 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:05.999709 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:05.999743 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:05.999793 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:05.999888 (+ 95us) server_negotiation.cc:299] Negotiation successful
0811 02:42:05.999917 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260811 02:42:06.000174 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 02:42:05.996996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.2.251.62:34509)
0811 02:42:05.997122 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 02:42:05.997124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 02:42:05.997133 (+ 9us) server_negotiation.cc:407] Connection header received
0811 02:42:05.997274 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 02:42:05.997276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 02:42:05.997339 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 02:42:05.997424 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 02:42:05.997529 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.998099 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.999127 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 02:42:05.999445 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 02:42:05.999512 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 02:42:05.999703 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 02:42:05.999827 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 02:42:05.999884 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 02:42:06.000069 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 02:42:06.000097 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260811 02:42:06.001909 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8549
I20260811 02:42:06.022959 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8679
I20260811 02:42:06.043188 3052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRZ17kF/build/release/bin/kudu with pid 8807
2026-08-11T02:42:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3159 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6250 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (294735 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST